[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:54:13 dist-test-slave-xzjr krb5kdc[6269](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:54:13 dist-test-slave-xzjr krb5kdc[6269](info): set up 2 sockets
Sep 25 04:54:13 dist-test-slave-xzjr krb5kdc[6269](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 04:54:14.822110 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.001s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 04:54:14 dist-test-slave-xzjr krb5kdc[6269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:54:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:54:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.29.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
I20260925 04:54:15.009277 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:45849
--webserver_interface=127.6.29.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:38919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:45849
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:54:15.342621 6282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:54:15.342959 6282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:54:15.343120 6282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:54:15.350585 6282 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:54:15.350713 6282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:54:15.350802 6282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:54:15.350903 6282 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:54:15.350979 6282 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:54:15.361205 6282 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:38919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:45849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:45849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:54:15.362695 6282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:54:15.364104 6282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:54:15.376775 6287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:15.376739 6290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:15.376825 6288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:54:15.377072 6282 server_base.cc:1034] running on GCE node
I20260925 04:54:15.378296 6282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:54:15.379961 6282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:54:15.381232 6282 hybrid_clock.cc:648] HybridClock initialized: now 1790312055381160 us; error 66 us; skew 500 ppm
I20260925 04:54:15.381621 6282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:54:15 dist-test-slave-xzjr krb5kdc[6269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312055, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:54:15.385097 6282 init.cc:377] Logged in from keytab as kudu/127.6.29.190@KRBTEST.COM (short username kudu)
I20260925 04:54:15.386701 6282 webserver.cc:460] Webserver started at http://127.6.29.190:39097/ using document root <none> and password file <none>
I20260925 04:54:15.387462 6282 fs_manager.cc:359] Metadata directory not provided
I20260925 04:54:15.387611 6282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:54:15.387991 6282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:54:15.391136 6282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "35141956babb4da7bcae6cfcf3e227e9"
format_stamp: "Formatted at 2026-09-25 04:54:15 on dist-test-slave-xzjr"
server_key: "8a4f9349f24e82902486ca67d991316a"
server_key_iv: "91f8dd1cebd573ad13c3926d310ee66e"
server_key_version: "encryptionkey@0"
I20260925 04:54:15.392074 6282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "35141956babb4da7bcae6cfcf3e227e9"
format_stamp: "Formatted at 2026-09-25 04:54:15 on dist-test-slave-xzjr"
server_key: "8a4f9349f24e82902486ca67d991316a"
server_key_iv: "91f8dd1cebd573ad13c3926d310ee66e"
server_key_version: "encryptionkey@0"
I20260925 04:54:15.397907 6282 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 04:54:15.401952 6297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:15.403633 6282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:54:15.403962 6282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "35141956babb4da7bcae6cfcf3e227e9"
format_stamp: "Formatted at 2026-09-25 04:54:15 on dist-test-slave-xzjr"
server_key: "8a4f9349f24e82902486ca67d991316a"
server_key_iv: "91f8dd1cebd573ad13c3926d310ee66e"
server_key_version: "encryptionkey@0"
I20260925 04:54:15.404197 6282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:54:15.435124 6282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:54:15.440029 6282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:54:15.440344 6282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:54:15.459887 6282 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:45849
I20260925 04:54:15.459905 6348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:45849 every 8 connection(s)
I20260925 04:54:15.461371 6282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:54:15.465615 6349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:54:15.467761 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 6282
I20260925 04:54:15.468140 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 04:54:15.470059 6262 external_mini_cluster.cc:1428] Setting key a065b963d864a8ba0eace04df3bb1b40
I20260925 04:54:15.477293 6349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35141956babb4da7bcae6cfcf3e227e9: Bootstrap starting.
I20260925 04:54:15.480600 6349 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35141956babb4da7bcae6cfcf3e227e9: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:15.481727 6349 log.cc:824] T 00000000000000000000000000000000 P 35141956babb4da7bcae6cfcf3e227e9: Log is configured to *not* fsync() on all Append() calls
I20260925 04:54:15.485180 6349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35141956babb4da7bcae6cfcf3e227e9: No bootstrap required, opened a new log
I20260925 04:54:15.494357 6349 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35141956babb4da7bcae6cfcf3e227e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35141956babb4da7bcae6cfcf3e227e9" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 45849 } }
I20260925 04:54:15.494810 6349 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35141956babb4da7bcae6cfcf3e227e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:15.494932 6349 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35141956babb4da7bcae6cfcf3e227e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35141956babb4da7bcae6cfcf3e227e9, State: Initialized, Role: FOLLOWER
I20260925 04:54:15.495657 6349 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35141956babb4da7bcae6cfcf3e227e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35141956babb4da7bcae6cfcf3e227e9" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 45849 } }
I20260925 04:54:15.495991 6349 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35141956babb4da7bcae6cfcf3e227e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:54:15.496136 6349 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35141956babb4da7bcae6cfcf3e227e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:54:15.496335 6349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35141956babb4da7bcae6cfcf3e227e9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:15.498971 6349 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35141956babb4da7bcae6cfcf3e227e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35141956babb4da7bcae6cfcf3e227e9" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 45849 } }
I20260925 04:54:15.499547 6349 leader_election.cc:302] T 00000000000000000000000000000000 P 35141956babb4da7bcae6cfcf3e227e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35141956babb4da7bcae6cfcf3e227e9; no voters:
I20260925 04:54:15.500277 6349 leader_election.cc:288] T 00000000000000000000000000000000 P 35141956babb4da7bcae6cfcf3e227e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:54:15.500447 6354 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35141956babb4da7bcae6cfcf3e227e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:54:15.501585 6354 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35141956babb4da7bcae6cfcf3e227e9 [term 1 LEADER]: Becoming Leader. State: Replica: 35141956babb4da7bcae6cfcf3e227e9, State: Running, Role: LEADER
Sep 25 04:54:15 dist-test-slave-xzjr krb5kdc[6269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:54:15.502146 6354 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35141956babb4da7bcae6cfcf3e227e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35141956babb4da7bcae6cfcf3e227e9" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 45849 } }
I20260925 04:54:15.502899 6349 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35141956babb4da7bcae6cfcf3e227e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:54:15.505163 6355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35141956babb4da7bcae6cfcf3e227e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35141956babb4da7bcae6cfcf3e227e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35141956babb4da7bcae6cfcf3e227e9" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 45849 } } }
I20260925 04:54:15.505565 6355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35141956babb4da7bcae6cfcf3e227e9 [sys.catalog]: This master's current role is: LEADER
I20260925 04:54:15.506291 6356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35141956babb4da7bcae6cfcf3e227e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35141956babb4da7bcae6cfcf3e227e9. Latest consensus state: current_term: 1 leader_uuid: "35141956babb4da7bcae6cfcf3e227e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35141956babb4da7bcae6cfcf3e227e9" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 45849 } } }
I20260925 04:54:15.506705 6356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35141956babb4da7bcae6cfcf3e227e9 [sys.catalog]: This master's current role is: LEADER
I20260925 04:54:15.507062 6359 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:54:15.516698 6351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:15.474304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55984 (local address 127.6.29.190:45849)
0925 04:54:15.475394 (+ 1090us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:15.475412 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:15.476869 (+ 1457us) server_negotiation.cc:407] Connection header received
0925 04:54:15.486458 (+ 9589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:15.486491 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:15.487050 (+ 559us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:15.487573 (+ 523us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:15.490633 (+ 3060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:15.492824 (+ 2191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:15.494725 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:15.495821 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:15.496034 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:15.503888 (+ 7854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:15.503904 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:15.503929 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:15.503992 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:15.509454 (+ 5462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:15.511094 (+ 1640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:15.511109 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:15.511122 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:15.511432 (+ 310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:15.512251 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:15.512256 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:15.512259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:15.513043 (+ 784us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:15.513368 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:15.514617 (+ 1249us) server_negotiation.cc:299] Negotiation successful
0925 04:54:15.515174 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":801,"thread_start_us":515,"threads_started":1}
I20260925 04:54:15.518648 6359 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:54:15.530637 6359 catalog_manager.cc:1347] Generated new cluster ID: 9bb0fa7049c0416788e65d8413ea6786
I20260925 04:54:15.530777 6359 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:54:15.544873 6359 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:54:15.546844 6359 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:54:15.554996 6359 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35141956babb4da7bcae6cfcf3e227e9: Generated new TSK 0
I20260925 04:54:15.556123 6359 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.29.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab.
Entry for principal kudu/127.6.29.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab.
I20260925 04:54:15.609089 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.129:0
--local_ip_for_outbound_sockets=127.6.29.129
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:45849
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:38919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:54:15.962555 6375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:54:15.962989 6375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:54:15.963155 6375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:54:15.970654 6375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:54:15.970773 6375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:54:15.970889 6375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.129
I20260925 04:54:15.981016 6375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:38919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:45849
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.129
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:54:15.982311 6375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:54:15.983735 6375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:54:15.997225 6380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:15.997253 6383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:15.997308 6381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:54:15.998584 6375 server_base.cc:1034] running on GCE node
I20260925 04:54:15.999256 6375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:54:16.000368 6375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:54:16.001616 6375 hybrid_clock.cc:648] HybridClock initialized: now 1790312056001584 us; error 45 us; skew 500 ppm
I20260925 04:54:16.001969 6375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:54:16 dist-test-slave-xzjr krb5kdc[6269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312056, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:54:16.005563 6375 init.cc:377] Logged in from keytab as kudu/127.6.29.129@KRBTEST.COM (short username kudu)
I20260925 04:54:16.007493 6375 webserver.cc:460] Webserver started at http://127.6.29.129:44725/ using document root <none> and password file <none>
I20260925 04:54:16.008378 6375 fs_manager.cc:359] Metadata directory not provided
I20260925 04:54:16.008530 6375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:54:16.008934 6375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:54:16.012285 6375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-0/data/instance:
uuid: "825d8805c19c409e9a639cc83fb807f9"
format_stamp: "Formatted at 2026-09-25 04:54:16 on dist-test-slave-xzjr"
server_key: "46ca4c7da5f4158993cab3ec70908928"
server_key_iv: "4c36c130283c74b6f48be9274f382a4c"
server_key_version: "encryptionkey@0"
I20260925 04:54:16.013118 6375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance:
uuid: "825d8805c19c409e9a639cc83fb807f9"
format_stamp: "Formatted at 2026-09-25 04:54:16 on dist-test-slave-xzjr"
server_key: "46ca4c7da5f4158993cab3ec70908928"
server_key_iv: "4c36c130283c74b6f48be9274f382a4c"
server_key_version: "encryptionkey@0"
I20260925 04:54:16.018703 6375 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 04:54:16.022614 6390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:16.024246 6375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:54:16.024417 6375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-0/wal
uuid: "825d8805c19c409e9a639cc83fb807f9"
format_stamp: "Formatted at 2026-09-25 04:54:16 on dist-test-slave-xzjr"
server_key: "46ca4c7da5f4158993cab3ec70908928"
server_key_iv: "4c36c130283c74b6f48be9274f382a4c"
server_key_version: "encryptionkey@0"
I20260925 04:54:16.024667 6375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:54:16.059980 6375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:54:16.064771 6375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:54:16.065075 6375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:54:16.066076 6375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:54:16.067494 6375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:54:16.067570 6375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:16.067699 6375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:54:16.067795 6375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:16.103288 6375 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.129:39821
I20260925 04:54:16.103336 6502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.129:39821 every 8 connection(s)
I20260925 04:54:16.104715 6375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
I20260925 04:54:16.109550 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 6375
I20260925 04:54:16.109773 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance
I20260925 04:54:16.110369 6262 external_mini_cluster.cc:1428] Setting key 6ce066578fde3fa3b9e099c65abaa302
Sep 25 04:54:16 dist-test-slave-xzjr krb5kdc[6269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312056, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.29.130@KRBTEST.COM; defaulting to no policy
I20260925 04:54:16.128327 6505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:16.107441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:52809 (local address 127.6.29.190:45849)
0925 04:54:16.107953 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:16.107959 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:16.109179 (+ 1220us) server_negotiation.cc:407] Connection header received
0925 04:54:16.111453 (+ 2274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:16.111457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:16.111549 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:16.111659 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:16.113554 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:16.114289 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:16.115970 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:16.116665 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:16.116832 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:16.121268 (+ 4436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:16.121273 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:16.121276 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:16.121311 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:16.124746 (+ 3435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:16.125628 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:16.125635 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:16.125639 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:16.125797 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:16.126343 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:16.126350 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:16.126357 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:16.126745 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:16.126912 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:16.127879 (+ 967us) server_negotiation.cc:299] Negotiation successful
0925 04:54:16.128053 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":109,"server-negotiator.queue_time_us":381,"thread_start_us":244,"threads_started":1}
I20260925 04:54:16.129575 6506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:16.108165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45849 (local address 127.6.29.129:52809)
0925 04:54:16.109018 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:16.109065 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:16.110887 (+ 1822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:16.112028 (+ 1141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:16.112040 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:16.112770 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:54:16.113381 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:16.113399 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:16.114467 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:16.114471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:16.115726 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:16.115738 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:16.116952 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:16.116960 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:16.117226 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:16.118023 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:54:16.118066 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:54:16.121012 (+ 2946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:54:16.124949 (+ 3937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:16.124957 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:16.124973 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:16.125471 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:16.125943 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:16.125948 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:16.125950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:16.126202 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:16.126925 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:54:16.126936 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:54:16.127378 (+ 442us) client_negotiation.cc:769] Sending connection context
0925 04:54:16.127677 (+ 299us) client_negotiation.cc:240] Negotiation successful
0925 04:54:16.128195 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":316,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.130@KRBTEST.COM" created.
I20260925 04:54:16.134645 6503 heartbeater.cc:347] Connected to a master server at 127.6.29.190:45849
I20260925 04:54:16.135059 6503 heartbeater.cc:464] Registering TS with master...
I20260925 04:54:16.136045 6503 heartbeater.cc:511] Master 127.6.29.190:45849 requested a full tablet report, sending...
I20260925 04:54:16.138396 6314 ts_manager.cc:194] Registered new tserver with Master: 825d8805c19c409e9a639cc83fb807f9 (127.6.29.129:39821)
I20260925 04:54:16.140105 6314 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.129@KRBTEST.COM'} at 127.6.29.129:52809
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab.
Entry for principal kudu/127.6.29.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab.
I20260925 04:54:16.157030 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.130:0
--local_ip_for_outbound_sockets=127.6.29.130
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:45849
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:38919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:54:16.489260 6509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:54:16.489607 6509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:54:16.489799 6509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:54:16.497458 6509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:54:16.497628 6509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:54:16.497808 6509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.130
I20260925 04:54:16.508222 6509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:38919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:45849
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.130
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:54:16.509717 6509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:54:16.511122 6509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:54:16.524888 6514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:16.524853 6515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:16.524820 6517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:54:16.526553 6509 server_base.cc:1034] running on GCE node
I20260925 04:54:16.527309 6509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:54:16.528514 6509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:54:16.529742 6509 hybrid_clock.cc:648] HybridClock initialized: now 1790312056529658 us; error 96 us; skew 500 ppm
I20260925 04:54:16.530112 6509 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:54:16 dist-test-slave-xzjr krb5kdc[6269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312056, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:54:16.533468 6509 init.cc:377] Logged in from keytab as kudu/127.6.29.130@KRBTEST.COM (short username kudu)
I20260925 04:54:16.535157 6509 webserver.cc:460] Webserver started at http://127.6.29.130:35725/ using document root <none> and password file <none>
I20260925 04:54:16.535960 6509 fs_manager.cc:359] Metadata directory not provided
I20260925 04:54:16.536108 6509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:54:16.536448 6509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:54:16.539470 6509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-1/data/instance:
uuid: "fb26c4ebe01a407d9e4bafca4c88f196"
format_stamp: "Formatted at 2026-09-25 04:54:16 on dist-test-slave-xzjr"
server_key: "23d07c7dbbbe88406447377c3fe55f1f"
server_key_iv: "0b6697efe43b0ffa9ac3cbbcfc792ac0"
server_key_version: "encryptionkey@0"
I20260925 04:54:16.540316 6509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance:
uuid: "fb26c4ebe01a407d9e4bafca4c88f196"
format_stamp: "Formatted at 2026-09-25 04:54:16 on dist-test-slave-xzjr"
server_key: "23d07c7dbbbe88406447377c3fe55f1f"
server_key_iv: "0b6697efe43b0ffa9ac3cbbcfc792ac0"
server_key_version: "encryptionkey@0"
I20260925 04:54:16.545821 6509 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 04:54:16.549479 6524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:16.550962 6509 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 04:54:16.551151 6509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-1/wal
uuid: "fb26c4ebe01a407d9e4bafca4c88f196"
format_stamp: "Formatted at 2026-09-25 04:54:16 on dist-test-slave-xzjr"
server_key: "23d07c7dbbbe88406447377c3fe55f1f"
server_key_iv: "0b6697efe43b0ffa9ac3cbbcfc792ac0"
server_key_version: "encryptionkey@0"
I20260925 04:54:16.551381 6509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:54:16.566310 6509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:54:16.570693 6509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:54:16.571017 6509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:54:16.572067 6509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:54:16.573597 6509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:54:16.573699 6509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:16.573853 6509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:54:16.573930 6509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:16.608238 6509 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.130:44231
I20260925 04:54:16.608286 6636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.130:44231 every 8 connection(s)
I20260925 04:54:16.609601 6509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
I20260925 04:54:16.614337 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 6509
I20260925 04:54:16.614547 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance
I20260925 04:54:16.614977 6262 external_mini_cluster.cc:1428] Setting key 09fa56579194a26a4e6d1d5615cf7535
Sep 25 04:54:16 dist-test-slave-xzjr krb5kdc[6269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312056, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.29.131@KRBTEST.COM; defaulting to no policy
I20260925 04:54:16.631731 6505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:16.612371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:52427 (local address 127.6.29.190:45849)
0925 04:54:16.612549 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:16.612554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:16.613684 (+ 1130us) server_negotiation.cc:407] Connection header received
0925 04:54:16.615831 (+ 2147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:16.615837 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:16.615937 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:16.616051 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:16.617968 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:16.619214 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:16.620886 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:16.621579 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:16.621680 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:16.625723 (+ 4043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:16.625729 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:16.625732 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:16.625767 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:16.628603 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:16.629421 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:16.629428 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:16.629431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:16.629575 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:16.630019 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:16.630025 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:16.630029 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:16.630300 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:16.630459 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:16.631157 (+ 698us) server_negotiation.cc:299] Negotiation successful
0925 04:54:16.631388 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 04:54:16.632879 6639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:16.612775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45849 (local address 127.6.29.130:52427)
0925 04:54:16.613514 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:16.613551 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:16.615450 (+ 1899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:16.616340 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:16.616352 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:16.617148 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:54:16.617782 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:16.617800 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:16.619393 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:16.619397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:16.620739 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:16.620751 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:16.621832 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:16.621842 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:16.622215 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:16.623112 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:54:16.623136 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:54:16.625468 (+ 2332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:54:16.628758 (+ 3290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:16.628770 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:16.628784 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:16.629298 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:16.629695 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:16.629700 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:16.629705 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:16.629903 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:16.630442 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:54:16.630451 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:54:16.630805 (+ 354us) client_negotiation.cc:769] Sending connection context
0925 04:54:16.631073 (+ 268us) client_negotiation.cc:240] Negotiation successful
0925 04:54:16.631640 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":293,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.131@KRBTEST.COM" created.
I20260925 04:54:16.634650 6637 heartbeater.cc:347] Connected to a master server at 127.6.29.190:45849
I20260925 04:54:16.634966 6637 heartbeater.cc:464] Registering TS with master...
I20260925 04:54:16.635691 6637 heartbeater.cc:511] Master 127.6.29.190:45849 requested a full tablet report, sending...
I20260925 04:54:16.637285 6314 ts_manager.cc:194] Registered new tserver with Master: fb26c4ebe01a407d9e4bafca4c88f196 (127.6.29.130:44231)
I20260925 04:54:16.638088 6314 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.130@KRBTEST.COM'} at 127.6.29.130:52427
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab.
Entry for principal kudu/127.6.29.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab.
I20260925 04:54:16.657545 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.131:0
--local_ip_for_outbound_sockets=127.6.29.131
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:45849
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:38919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:54:16.989928 6642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:54:16.990217 6642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:54:16.990314 6642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:54:16.997493 6642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:54:16.997607 6642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:54:16.997725 6642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.131
I20260925 04:54:17.007709 6642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:38919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:45849
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.131
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:54:17.009248 6642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:54:17.010555 6642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:54:17.024109 6647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:17.024082 6648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:17.024139 6650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:54:17.024878 6642 server_base.cc:1034] running on GCE node
I20260925 04:54:17.025490 6642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:54:17.026573 6642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:54:17.027865 6642 hybrid_clock.cc:648] HybridClock initialized: now 1790312057027814 us; error 49 us; skew 500 ppm
I20260925 04:54:17.028191 6642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:54:17 dist-test-slave-xzjr krb5kdc[6269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312057, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:54:17.031910 6642 init.cc:377] Logged in from keytab as kudu/127.6.29.131@KRBTEST.COM (short username kudu)
I20260925 04:54:17.033593 6642 webserver.cc:460] Webserver started at http://127.6.29.131:39367/ using document root <none> and password file <none>
I20260925 04:54:17.034332 6642 fs_manager.cc:359] Metadata directory not provided
I20260925 04:54:17.034482 6642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:54:17.034816 6642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:54:17.037883 6642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-2/data/instance:
uuid: "63bc38f43b3147efbe21c248587a745d"
format_stamp: "Formatted at 2026-09-25 04:54:17 on dist-test-slave-xzjr"
server_key: "2d72888da24fbfeef3311014f5c0ff27"
server_key_iv: "fdcd1585216d19b1da51acb5e506ac61"
server_key_version: "encryptionkey@0"
I20260925 04:54:17.038668 6642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance:
uuid: "63bc38f43b3147efbe21c248587a745d"
format_stamp: "Formatted at 2026-09-25 04:54:17 on dist-test-slave-xzjr"
server_key: "2d72888da24fbfeef3311014f5c0ff27"
server_key_iv: "fdcd1585216d19b1da51acb5e506ac61"
server_key_version: "encryptionkey@0"
I20260925 04:54:17.044490 6642 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 04:54:17.048243 6657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:17.049716 6642 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:54:17.050006 6642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-2/wal
uuid: "63bc38f43b3147efbe21c248587a745d"
format_stamp: "Formatted at 2026-09-25 04:54:17 on dist-test-slave-xzjr"
server_key: "2d72888da24fbfeef3311014f5c0ff27"
server_key_iv: "fdcd1585216d19b1da51acb5e506ac61"
server_key_version: "encryptionkey@0"
I20260925 04:54:17.050235 6642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:54:17.085001 6642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:54:17.089716 6642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:54:17.090039 6642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:54:17.091073 6642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:54:17.092504 6642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:54:17.092608 6642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:17.092773 6642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:54:17.092851 6642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:17.129957 6642 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.131:32935
I20260925 04:54:17.129979 6769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.131:32935 every 8 connection(s)
I20260925 04:54:17.131284 6642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
I20260925 04:54:17.135051 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 6642
I20260925 04:54:17.135253 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance
I20260925 04:54:17.135819 6262 external_mini_cluster.cc:1428] Setting key 0758a2a7886595c4d91b3a3edfead50d
I20260925 04:54:17.144213 6503 heartbeater.cc:503] Master 127.6.29.190:45849 was elected leader, sending a full tablet report...
Sep 25 04:54:17 dist-test-slave-xzjr krb5kdc[6269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312057, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:54:17.153069 6772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:17.133918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:36193 (local address 127.6.29.190:45849)
0925 04:54:17.134364 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:17.134370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:17.135586 (+ 1216us) server_negotiation.cc:407] Connection header received
0925 04:54:17.137954 (+ 2368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:17.137958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:17.138036 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:17.138139 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:17.140022 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:17.140818 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:17.142485 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:17.142931 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:17.143055 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:17.147138 (+ 4083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:17.147142 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:17.147145 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:17.147176 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:17.149942 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:17.150690 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:17.150703 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:17.150705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:17.150834 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:17.151309 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:17.151321 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:17.151323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:17.151586 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:17.151741 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:17.152490 (+ 749us) server_negotiation.cc:299] Negotiation successful
0925 04:54:17.152698 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":211,"threads_started":1}
I20260925 04:54:17.154287 6773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:17.134567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45849 (local address 127.6.29.131:36193)
0925 04:54:17.135383 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:17.135451 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:17.137611 (+ 2160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:17.138446 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:17.138458 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:17.139070 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:54:17.139823 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:17.139849 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:17.141003 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:17.141007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:17.142299 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:17.142311 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:17.143157 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:17.143161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:17.143396 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:17.144249 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:54:17.144288 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:54:17.146924 (+ 2636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:54:17.150095 (+ 3171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:17.150121 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:17.150135 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:17.150555 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:17.150963 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:17.150968 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:17.150978 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:17.151190 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:17.151728 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:54:17.151736 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:54:17.152123 (+ 387us) client_negotiation.cc:769] Sending connection context
0925 04:54:17.152385 (+ 262us) client_negotiation.cc:240] Negotiation successful
0925 04:54:17.152921 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":308,"threads_started":1}
I20260925 04:54:17.155864 6770 heartbeater.cc:347] Connected to a master server at 127.6.29.190:45849
I20260925 04:54:17.156162 6770 heartbeater.cc:464] Registering TS with master...
I20260925 04:54:17.156833 6770 heartbeater.cc:511] Master 127.6.29.190:45849 requested a full tablet report, sending...
I20260925 04:54:17.158125 6314 ts_manager.cc:194] Registered new tserver with Master: 63bc38f43b3147efbe21c248587a745d (127.6.29.131:32935)
I20260925 04:54:17.159003 6314 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.131@KRBTEST.COM'} at 127.6.29.131:36193
I20260925 04:54:17.164455 6262 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 04:54:17 dist-test-slave-xzjr krb5kdc[6269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312057, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 04:54:17 dist-test-slave-xzjr krb5kdc[6269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312057, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.29.129@KRBTEST.COM
I20260925 04:54:17.209975 6778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:17.193676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35816 (local address 127.6.29.129:39821)
0925 04:54:17.194472 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:17.194480 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:17.194510 (+ 30us) server_negotiation.cc:407] Connection header received
0925 04:54:17.194568 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:17.194574 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:17.194814 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:17.194952 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:17.196499 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:17.197513 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:17.198704 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:17.199357 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:17.199461 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:17.203566 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:17.203578 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:17.203593 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:17.203632 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:17.206681 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:17.207628 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:17.207642 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:17.207647 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:17.207858 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:17.208274 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:17.208280 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:17.208288 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:17.208710 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:17.209037 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:17.209343 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 04:54:17.209577 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":511,"threads_started":1}
W20260925 04:54:17.211301 6418 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35816
Sep 25 04:54:17 dist-test-slave-xzjr krb5kdc[6269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 04:54:17 dist-test-slave-xzjr krb5kdc[6269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.129@KRBTEST.COM
I20260925 04:54:17.256688 6778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:17.242262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35820 (local address 127.6.29.129:39821)
0925 04:54:17.242513 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:17.242520 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:17.242810 (+ 290us) server_negotiation.cc:407] Connection header received
0925 04:54:17.243069 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:17.243074 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:17.243155 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:17.243264 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:17.244686 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:17.245369 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:17.246602 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:17.247016 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:17.247152 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:17.251214 (+ 4062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:17.251219 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:17.251223 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:17.251259 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:17.253983 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:17.254690 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:17.254699 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:17.254703 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:17.254904 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:17.255285 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:17.255290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:17.255294 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:17.255564 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:17.255825 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:17.256137 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 04:54:17.256356 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 04:54:17.258183 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 6375
I20260925 04:54:17.270749 6778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:17.259139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35836 (local address 127.6.29.129:39821)
0925 04:54:17.259332 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:17.259339 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:17.259675 (+ 336us) server_negotiation.cc:407] Connection header received
0925 04:54:17.259930 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:17.259941 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:17.259998 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:17.260099 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:17.261225 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:17.261950 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:17.263184 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:17.263617 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:17.263753 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:17.265935 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:17.265939 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:17.265943 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:17.265970 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:17.268304 (+ 2334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:17.268911 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:17.268919 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:17.268927 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:17.269050 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:17.269406 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:17.269413 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:17.269417 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:17.269668 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:17.269919 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:17.270181 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 04:54:17.270381 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 04:54:17.271420 6498 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:54:17.355546 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 6375
I20260925 04:54:17.372304 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 6509
Sep 25 04:54:17 dist-test-slave-xzjr krb5kdc[6269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.130@KRBTEST.COM
I20260925 04:54:17.388000 6785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:17.373215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33368 (local address 127.6.29.130:44231)
0925 04:54:17.373713 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:17.373721 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:17.373745 (+ 24us) server_negotiation.cc:407] Connection header received
0925 04:54:17.373804 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:17.373811 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:17.374031 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:17.374203 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:17.375382 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:17.376447 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:17.377657 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:17.378201 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:17.378340 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:17.382424 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:17.382435 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:17.382451 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:17.382483 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:17.385227 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:17.385875 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:17.385883 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:17.385885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:17.386007 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:17.386393 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:17.386400 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:17.386403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:17.386724 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:17.387055 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:17.387332 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 04:54:17.387596 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":268,"threads_started":1}
I20260925 04:54:17.389348 6632 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:54:17.476305 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 6509
I20260925 04:54:17.498955 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 6642
Sep 25 04:54:17 dist-test-slave-xzjr krb5kdc[6269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.131@KRBTEST.COM
I20260925 04:54:17.515168 6787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:17.499944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55828 (local address 127.6.29.131:32935)
0925 04:54:17.500545 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:17.500552 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:17.500582 (+ 30us) server_negotiation.cc:407] Connection header received
0925 04:54:17.500640 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:17.500647 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:17.500902 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:17.501085 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:17.502316 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:17.503398 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:17.504762 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:17.505319 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:17.505470 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:17.509686 (+ 4216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:17.509697 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:17.509712 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:17.509784 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:17.512731 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:17.513382 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:17.513388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:17.513391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:17.513508 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:17.513852 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:17.513855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:17.513857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:17.514174 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:17.514402 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:17.514659 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 04:54:17.514889 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":279,"threads_started":1}
I20260925 04:54:17.516520 6765 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:54:17.600140 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 6642
I20260925 04:54:17.616914 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 6282
I20260925 04:54:17.618076 6344 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:54:17.727841 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 6282
2026-09-25T04:54:17Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4029 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:54:17 dist-test-slave-xzjr krb5kdc[6792](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:54:17 dist-test-slave-xzjr krb5kdc[6792](info): set up 2 sockets
Sep 25 04:54:17 dist-test-slave-xzjr krb5kdc[6792](info): commencing operation
krb5kdc: starting...
W20260925 04:54:18.828387 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.009s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 04:54:18 dist-test-slave-xzjr krb5kdc[6792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:54:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:54:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.29.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
I20260925 04:54:19.020169 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:46465
--webserver_interface=127.6.29.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:33575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:46465
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:54:19.351624 6805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:54:19.352007 6805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:54:19.352197 6805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:54:19.359537 6805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:54:19.359650 6805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:54:19.359740 6805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:54:19.359859 6805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:54:19.359957 6805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:54:19.370023 6805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:33575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:46465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:46465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:54:19.371423 6805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:54:19.372916 6805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:54:19.385133 6810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:19.385089 6811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:19.385172 6813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:54:19.386404 6805 server_base.cc:1034] running on GCE node
I20260925 04:54:19.387221 6805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:54:19.388784 6805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:54:19.389987 6805 hybrid_clock.cc:648] HybridClock initialized: now 1790312059389921 us; error 59 us; skew 500 ppm
I20260925 04:54:19.390348 6805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:54:19 dist-test-slave-xzjr krb5kdc[6792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312059, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:54:19.397307 6805 init.cc:377] Logged in from keytab as kudu/127.6.29.190@KRBTEST.COM (short username kudu)
I20260925 04:54:19.399014 6805 webserver.cc:460] Webserver started at http://127.6.29.190:44345/ using document root <none> and password file <none>
I20260925 04:54:19.399750 6805 fs_manager.cc:359] Metadata directory not provided
I20260925 04:54:19.399955 6805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:54:19.400277 6805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:54:19.403260 6805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "1b860a06f339458a993095907ed58cad"
format_stamp: "Formatted at 2026-09-25 04:54:19 on dist-test-slave-xzjr"
server_key: "e2f601838e8899c735f9475320ebd5f0"
server_key_iv: "1f6b5457b3b9fb226b42e0fefa0bd1bc"
server_key_version: "encryptionkey@0"
I20260925 04:54:19.404083 6805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "1b860a06f339458a993095907ed58cad"
format_stamp: "Formatted at 2026-09-25 04:54:19 on dist-test-slave-xzjr"
server_key: "e2f601838e8899c735f9475320ebd5f0"
server_key_iv: "1f6b5457b3b9fb226b42e0fefa0bd1bc"
server_key_version: "encryptionkey@0"
I20260925 04:54:19.409529 6805 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 04:54:19.413151 6820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:19.414597 6805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:54:19.414795 6805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "1b860a06f339458a993095907ed58cad"
format_stamp: "Formatted at 2026-09-25 04:54:19 on dist-test-slave-xzjr"
server_key: "e2f601838e8899c735f9475320ebd5f0"
server_key_iv: "1f6b5457b3b9fb226b42e0fefa0bd1bc"
server_key_version: "encryptionkey@0"
I20260925 04:54:19.415002 6805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:54:19.441829 6805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:54:19.446197 6805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:54:19.446522 6805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:54:19.465345 6805 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:46465
I20260925 04:54:19.465382 6871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:46465 every 8 connection(s)
I20260925 04:54:19.466749 6805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:54:19.467942 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 6805
I20260925 04:54:19.468168 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 04:54:19.468968 6262 external_mini_cluster.cc:1428] Setting key c8dc2ba9a4a2b3ed1fd36d790ac1ffda
I20260925 04:54:19.471853 6872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 04:54:19 dist-test-slave-xzjr krb5kdc[6792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:54:19.485415 6872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b860a06f339458a993095907ed58cad: Bootstrap starting.
I20260925 04:54:19.488986 6872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b860a06f339458a993095907ed58cad: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:19.490074 6872 log.cc:824] T 00000000000000000000000000000000 P 1b860a06f339458a993095907ed58cad: Log is configured to *not* fsync() on all Append() calls
I20260925 04:54:19.492800 6875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:19.471145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53168 (local address 127.6.29.190:46465)
0925 04:54:19.471990 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:19.472006 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:19.472057 (+ 51us) server_negotiation.cc:407] Connection header received
0925 04:54:19.473682 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:19.473704 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:19.474110 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:19.474468 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:19.476279 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:19.477678 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:19.479522 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:19.480284 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:19.480435 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:19.485126 (+ 4691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:19.485135 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:19.485153 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:19.485195 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:19.488312 (+ 3117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:19.489007 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:19.489014 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:19.489020 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:19.489233 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:19.489593 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:19.489600 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:19.489603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:19.490152 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:19.490475 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:19.490997 (+ 522us) server_negotiation.cc:299] Negotiation successful
0925 04:54:19.491457 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":371,"threads_started":1}
I20260925 04:54:19.493477 6872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b860a06f339458a993095907ed58cad: No bootstrap required, opened a new log
I20260925 04:54:19.501353 6872 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b860a06f339458a993095907ed58cad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b860a06f339458a993095907ed58cad" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 46465 } }
I20260925 04:54:19.501736 6872 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b860a06f339458a993095907ed58cad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:19.501883 6872 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b860a06f339458a993095907ed58cad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b860a06f339458a993095907ed58cad, State: Initialized, Role: FOLLOWER
I20260925 04:54:19.502456 6872 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b860a06f339458a993095907ed58cad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b860a06f339458a993095907ed58cad" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 46465 } }
I20260925 04:54:19.502681 6872 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b860a06f339458a993095907ed58cad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:54:19.502805 6872 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b860a06f339458a993095907ed58cad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:54:19.502983 6872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b860a06f339458a993095907ed58cad [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:19.505343 6872 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b860a06f339458a993095907ed58cad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b860a06f339458a993095907ed58cad" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 46465 } }
I20260925 04:54:19.505889 6872 leader_election.cc:302] T 00000000000000000000000000000000 P 1b860a06f339458a993095907ed58cad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b860a06f339458a993095907ed58cad; no voters:
I20260925 04:54:19.506446 6872 leader_election.cc:288] T 00000000000000000000000000000000 P 1b860a06f339458a993095907ed58cad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:54:19.506608 6877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b860a06f339458a993095907ed58cad [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:54:19.507304 6877 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b860a06f339458a993095907ed58cad [term 1 LEADER]: Becoming Leader. State: Replica: 1b860a06f339458a993095907ed58cad, State: Running, Role: LEADER
I20260925 04:54:19.508037 6877 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b860a06f339458a993095907ed58cad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b860a06f339458a993095907ed58cad" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 46465 } }
I20260925 04:54:19.508657 6872 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b860a06f339458a993095907ed58cad [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:54:19.511144 6878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b860a06f339458a993095907ed58cad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b860a06f339458a993095907ed58cad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b860a06f339458a993095907ed58cad" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 46465 } } }
I20260925 04:54:19.512562 6878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b860a06f339458a993095907ed58cad [sys.catalog]: This master's current role is: LEADER
I20260925 04:54:19.511242 6879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b860a06f339458a993095907ed58cad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b860a06f339458a993095907ed58cad. Latest consensus state: current_term: 1 leader_uuid: "1b860a06f339458a993095907ed58cad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b860a06f339458a993095907ed58cad" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 46465 } } }
I20260925 04:54:19.513064 6879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b860a06f339458a993095907ed58cad [sys.catalog]: This master's current role is: LEADER
I20260925 04:54:19.514564 6885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:54:19.519870 6885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:54:19.528648 6885 catalog_manager.cc:1347] Generated new cluster ID: d69f791a94644813b367788deea0934f
I20260925 04:54:19.528771 6885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:54:19.541044 6885 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:54:19.542470 6885 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:54:19.565056 6885 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b860a06f339458a993095907ed58cad: Generated new TSK 0
I20260925 04:54:19.566013 6885 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.29.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab.
Entry for principal kudu/127.6.29.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab.
W20260925 04:54:20.677625 6262 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.6.29.129: real 1.102s user 0.004s sys 0.005s
I20260925 04:54:20.678684 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.129:0
--local_ip_for_outbound_sockets=127.6.29.129
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:46465
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:33575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:54:21.033851 6898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:54:21.034209 6898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:54:21.034307 6898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:54:21.042110 6898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:54:21.042243 6898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:54:21.042362 6898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.129
I20260925 04:54:21.053233 6898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:33575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:46465
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.129
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:54:21.054777 6898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:54:21.056280 6898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:54:21.070677 6903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:21.070600 6904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:21.070640 6906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:54:21.072002 6898 server_base.cc:1034] running on GCE node
I20260925 04:54:21.072628 6898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:54:21.073630 6898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:54:21.074855 6898 hybrid_clock.cc:648] HybridClock initialized: now 1790312061074823 us; error 50 us; skew 500 ppm
I20260925 04:54:21.075218 6898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:54:21 dist-test-slave-xzjr krb5kdc[6792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312061, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:54:21.078725 6898 init.cc:377] Logged in from keytab as kudu/127.6.29.129@KRBTEST.COM (short username kudu)
I20260925 04:54:21.080251 6898 webserver.cc:460] Webserver started at http://127.6.29.129:35883/ using document root <none> and password file <none>
I20260925 04:54:21.080977 6898 fs_manager.cc:359] Metadata directory not provided
I20260925 04:54:21.081110 6898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:54:21.081436 6898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:54:21.084452 6898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-0/data/instance:
uuid: "83a6bc29f6f74af7b182fd004cf414d5"
format_stamp: "Formatted at 2026-09-25 04:54:21 on dist-test-slave-xzjr"
server_key: "663b0736708f351ac9bebc97804d08d2"
server_key_iv: "62439aa66c594adca1152d02d0669ca4"
server_key_version: "encryptionkey@0"
I20260925 04:54:21.085290 6898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance:
uuid: "83a6bc29f6f74af7b182fd004cf414d5"
format_stamp: "Formatted at 2026-09-25 04:54:21 on dist-test-slave-xzjr"
server_key: "663b0736708f351ac9bebc97804d08d2"
server_key_iv: "62439aa66c594adca1152d02d0669ca4"
server_key_version: "encryptionkey@0"
I20260925 04:54:21.090891 6898 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 04:54:21.094614 6913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:21.096211 6898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:54:21.096413 6898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-0/wal
uuid: "83a6bc29f6f74af7b182fd004cf414d5"
format_stamp: "Formatted at 2026-09-25 04:54:21 on dist-test-slave-xzjr"
server_key: "663b0736708f351ac9bebc97804d08d2"
server_key_iv: "62439aa66c594adca1152d02d0669ca4"
server_key_version: "encryptionkey@0"
I20260925 04:54:21.096623 6898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:54:21.120393 6898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:54:21.125157 6898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:54:21.125495 6898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:54:21.126564 6898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:54:21.128023 6898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:54:21.128140 6898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:21.128294 6898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:54:21.128374 6898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:21.167673 6898 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.129:43935
I20260925 04:54:21.167776 7025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.129:43935 every 8 connection(s)
I20260925 04:54:21.169030 6898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
I20260925 04:54:21.178026 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 6898
I20260925 04:54:21.178234 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance
I20260925 04:54:21.178901 6262 external_mini_cluster.cc:1428] Setting key 4c112d1c5aa51f30e39496bdaa6722f8
Sep 25 04:54:21 dist-test-slave-xzjr krb5kdc[6792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312061, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:54:21.190716 7028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:21.171913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:53523 (local address 127.6.29.190:46465)
0925 04:54:21.172390 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:21.172396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:21.173492 (+ 1096us) server_negotiation.cc:407] Connection header received
0925 04:54:21.175352 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:21.175357 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:21.175423 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:21.175533 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:21.177380 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:21.178092 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:21.179941 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:21.180516 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:21.180647 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:21.184309 (+ 3662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:21.184319 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:21.184327 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:21.184385 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:21.187229 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:21.188243 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:21.188246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:21.188249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:21.188369 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:21.188869 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:21.188872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:21.188874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:21.189155 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:21.189280 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:21.190245 (+ 965us) server_negotiation.cc:299] Negotiation successful
0925 04:54:21.190434 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":219,"threads_started":1}
I20260925 04:54:21.192456 7029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:21.172567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:46465 (local address 127.6.29.129:53523)
0925 04:54:21.173326 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:21.173364 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:21.175032 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:21.175857 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:21.175871 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:21.176517 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:54:21.177164 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:21.177182 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:21.178273 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:21.178278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:21.179736 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:21.179750 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:21.180840 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:21.180848 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:21.181073 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:21.181793 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:54:21.181816 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:54:21.184066 (+ 2250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:54:21.187376 (+ 3310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:21.187388 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:21.187410 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:21.188118 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:21.188478 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:21.188486 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:21.188492 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:21.188739 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:21.189304 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:54:21.189316 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:54:21.189811 (+ 495us) client_negotiation.cc:769] Sending connection context
0925 04:54:21.190127 (+ 316us) client_negotiation.cc:240] Negotiation successful
0925 04:54:21.190741 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":308,"threads_started":1}
I20260925 04:54:21.194308 7026 heartbeater.cc:347] Connected to a master server at 127.6.29.190:46465
I20260925 04:54:21.194680 7026 heartbeater.cc:464] Registering TS with master...
I20260925 04:54:21.195465 7026 heartbeater.cc:511] Master 127.6.29.190:46465 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.29.130@KRBTEST.COM; defaulting to no policy
I20260925 04:54:21.197893 6837 ts_manager.cc:194] Registered new tserver with Master: 83a6bc29f6f74af7b182fd004cf414d5 (127.6.29.129:43935)
I20260925 04:54:21.199445 6837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.129@KRBTEST.COM'} at 127.6.29.129:53523
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.130@KRBTEST.COM" created.
I20260925 04:54:22.202713 7026 heartbeater.cc:503] Master 127.6.29.190:46465 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab.
Entry for principal kudu/127.6.29.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab.
W20260925 04:54:22.531695 6262 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.6.29.130: real 1.350s user 0.001s sys 0.009s
I20260925 04:54:22.533404 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.130:0
--local_ip_for_outbound_sockets=127.6.29.130
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:46465
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:33575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:54:22.872143 7032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:54:22.872512 7032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:54:22.872691 7032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:54:22.880218 7032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:54:22.880394 7032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:54:22.880586 7032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.130
I20260925 04:54:22.890594 7032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:33575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:46465
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.130
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:54:22.892114 7032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:54:22.893507 7032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:54:22.907034 7037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:22.907269 7038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:22.907519 7040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:54:22.908541 7032 server_base.cc:1034] running on GCE node
I20260925 04:54:22.909278 7032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:54:22.910385 7032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:54:22.911592 7032 hybrid_clock.cc:648] HybridClock initialized: now 1790312062911538 us; error 44 us; skew 500 ppm
I20260925 04:54:22.911988 7032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:54:22 dist-test-slave-xzjr krb5kdc[6792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312062, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:54:22.915366 7032 init.cc:377] Logged in from keytab as kudu/127.6.29.130@KRBTEST.COM (short username kudu)
I20260925 04:54:22.917096 7032 webserver.cc:460] Webserver started at http://127.6.29.130:43335/ using document root <none> and password file <none>
I20260925 04:54:22.917840 7032 fs_manager.cc:359] Metadata directory not provided
I20260925 04:54:22.917968 7032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:54:22.918296 7032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:54:22.921522 7032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-1/data/instance:
uuid: "6a6eb8c2877c4af0b6e7df8e3b2bc8e5"
format_stamp: "Formatted at 2026-09-25 04:54:22 on dist-test-slave-xzjr"
server_key: "cd1c3f3b86fedc701305f030d03c1fdf"
server_key_iv: "a52f062300cb80a296b011bb0e4754ae"
server_key_version: "encryptionkey@0"
I20260925 04:54:22.922430 7032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance:
uuid: "6a6eb8c2877c4af0b6e7df8e3b2bc8e5"
format_stamp: "Formatted at 2026-09-25 04:54:22 on dist-test-slave-xzjr"
server_key: "cd1c3f3b86fedc701305f030d03c1fdf"
server_key_iv: "a52f062300cb80a296b011bb0e4754ae"
server_key_version: "encryptionkey@0"
I20260925 04:54:22.928110 7032 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260925 04:54:22.931921 7047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:22.933369 7032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 04:54:22.933562 7032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-1/wal
uuid: "6a6eb8c2877c4af0b6e7df8e3b2bc8e5"
format_stamp: "Formatted at 2026-09-25 04:54:22 on dist-test-slave-xzjr"
server_key: "cd1c3f3b86fedc701305f030d03c1fdf"
server_key_iv: "a52f062300cb80a296b011bb0e4754ae"
server_key_version: "encryptionkey@0"
I20260925 04:54:22.933780 7032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:54:22.956460 7032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:54:22.962471 7032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:54:22.962738 7032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:54:22.963889 7032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:54:22.965238 7032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:54:22.965314 7032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:22.965448 7032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:54:22.965497 7032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:22.998505 7032 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.130:38321
I20260925 04:54:22.998523 7159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.130:38321 every 8 connection(s)
I20260925 04:54:22.999992 7032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
I20260925 04:54:23.001685 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 7032
I20260925 04:54:23.001922 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance
I20260925 04:54:23.002449 6262 external_mini_cluster.cc:1428] Setting key e7361511acd4f65a392fda1afa1635f5
Sep 25 04:54:23 dist-test-slave-xzjr krb5kdc[6792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312062, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.29.131@KRBTEST.COM; defaulting to no policy
I20260925 04:54:23.026628 7162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:23.002979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:56335 (local address 127.6.29.190:46465)
0925 04:54:23.003530 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:23.003536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:23.005273 (+ 1737us) server_negotiation.cc:407] Connection header received
0925 04:54:23.008244 (+ 2971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:23.008253 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:23.008350 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:23.008491 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:23.010743 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:23.012024 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:23.013787 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:23.014560 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:23.014708 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:23.018749 (+ 4041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:23.018756 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:23.018763 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:23.018809 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:23.022259 (+ 3450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:23.023249 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:23.023257 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:23.023261 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:23.023400 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:23.023907 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:23.023913 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:23.023917 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:23.024204 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:23.024368 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:23.025164 (+ 796us) server_negotiation.cc:299] Negotiation successful
0925 04:54:23.025395 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":299,"threads_started":1}
I20260925 04:54:23.027161 7163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:23.003661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:46465 (local address 127.6.29.130:56335)
0925 04:54:23.005082 (+ 1421us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:23.005132 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:23.007932 (+ 2800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:23.008847 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:23.008864 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:23.009839 (+ 975us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:54:23.010558 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:23.010578 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:23.012202 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:23.012207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:23.013558 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:23.013571 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:23.014825 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:23.014831 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:23.015047 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:23.016032 (+ 985us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:54:23.016059 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:54:23.018479 (+ 2420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:54:23.022449 (+ 3970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:23.022473 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:23.022490 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:23.023104 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:23.023524 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:23.023529 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:23.023533 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:23.023733 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:23.024342 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:54:23.024350 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:54:23.024733 (+ 383us) client_negotiation.cc:769] Sending connection context
0925 04:54:23.025068 (+ 335us) client_negotiation.cc:240] Negotiation successful
0925 04:54:23.025654 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":355,"threads_started":1}
I20260925 04:54:23.028784 7160 heartbeater.cc:347] Connected to a master server at 127.6.29.190:46465
I20260925 04:54:23.029074 7160 heartbeater.cc:464] Registering TS with master...
I20260925 04:54:23.029762 7160 heartbeater.cc:511] Master 127.6.29.190:46465 requested a full tablet report, sending...
I20260925 04:54:23.031340 6837 ts_manager.cc:194] Registered new tserver with Master: 6a6eb8c2877c4af0b6e7df8e3b2bc8e5 (127.6.29.130:38321)
I20260925 04:54:23.032280 6837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.130@KRBTEST.COM'} at 127.6.29.130:56335
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.131@KRBTEST.COM" created.
I20260925 04:54:24.035048 7160 heartbeater.cc:503] Master 127.6.29.190:46465 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab.
Entry for principal kudu/127.6.29.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab.
W20260925 04:54:24.350303 6262 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.6.29.131: real 1.345s user 0.004s sys 0.004s
I20260925 04:54:24.351066 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.131:0
--local_ip_for_outbound_sockets=127.6.29.131
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:46465
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:33575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:54:24.686059 7166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:54:24.686403 7166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:54:24.686578 7166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:54:24.694326 7166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:54:24.694511 7166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:54:24.694725 7166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.131
I20260925 04:54:24.705108 7166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:33575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:46465
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.131
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:54:24.706507 7166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:54:24.707979 7166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:54:24.721828 7171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:24.721807 7172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:24.721778 7174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:54:24.722623 7166 server_base.cc:1034] running on GCE node
I20260925 04:54:24.723224 7166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:54:24.724294 7166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:54:24.725502 7166 hybrid_clock.cc:648] HybridClock initialized: now 1790312064725453 us; error 40 us; skew 500 ppm
I20260925 04:54:24.725847 7166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:54:24 dist-test-slave-xzjr krb5kdc[6792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312064, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:54:24.733655 7166 init.cc:377] Logged in from keytab as kudu/127.6.29.131@KRBTEST.COM (short username kudu)
I20260925 04:54:24.735311 7166 webserver.cc:460] Webserver started at http://127.6.29.131:35271/ using document root <none> and password file <none>
I20260925 04:54:24.736111 7166 fs_manager.cc:359] Metadata directory not provided
I20260925 04:54:24.736265 7166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:54:24.736589 7166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:54:24.739892 7166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-2/data/instance:
uuid: "86247c2984bc451badd0a9eabac12a2c"
format_stamp: "Formatted at 2026-09-25 04:54:24 on dist-test-slave-xzjr"
server_key: "4e7ebb52dc1945c143f3372f753c8322"
server_key_iv: "aa059972597c98a7bae53be9b30b1b46"
server_key_version: "encryptionkey@0"
I20260925 04:54:24.740706 7166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance:
uuid: "86247c2984bc451badd0a9eabac12a2c"
format_stamp: "Formatted at 2026-09-25 04:54:24 on dist-test-slave-xzjr"
server_key: "4e7ebb52dc1945c143f3372f753c8322"
server_key_iv: "aa059972597c98a7bae53be9b30b1b46"
server_key_version: "encryptionkey@0"
I20260925 04:54:24.746173 7166 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 04:54:24.749993 7181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:24.751401 7166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:54:24.751605 7166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-2/wal
uuid: "86247c2984bc451badd0a9eabac12a2c"
format_stamp: "Formatted at 2026-09-25 04:54:24 on dist-test-slave-xzjr"
server_key: "4e7ebb52dc1945c143f3372f753c8322"
server_key_iv: "aa059972597c98a7bae53be9b30b1b46"
server_key_version: "encryptionkey@0"
I20260925 04:54:24.751848 7166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:54:24.776402 7166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:54:24.780869 7166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:54:24.781179 7166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:54:24.782183 7166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:54:24.783560 7166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:54:24.783663 7166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:24.783838 7166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:54:24.783915 7166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260925 04:54:24.817765 7166 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.131:37453
I20260925 04:54:24.817848 7293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.131:37453 every 8 connection(s)
I20260925 04:54:24.819267 7166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
I20260925 04:54:24.829169 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 7166
I20260925 04:54:24.829461 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance
I20260925 04:54:24.830058 6262 external_mini_cluster.cc:1428] Setting key 64549178f6336feb69d91d055f16a908
Sep 25 04:54:24 dist-test-slave-xzjr krb5kdc[6792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312064, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:54:24.840183 7296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:24.821960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:58055 (local address 127.6.29.190:46465)
0925 04:54:24.822446 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:24.822453 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:24.823451 (+ 998us) server_negotiation.cc:407] Connection header received
0925 04:54:24.825341 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:24.825345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:24.825426 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:24.825526 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:24.827028 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:24.827866 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:24.829999 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:24.830458 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:24.830558 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:24.834608 (+ 4050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:24.834614 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:24.834618 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:24.834650 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:24.837166 (+ 2516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:24.837904 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:24.837909 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:24.837912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:24.838037 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:24.838619 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:24.838623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:24.838625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:24.838940 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:24.839058 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:24.839686 (+ 628us) server_negotiation.cc:299] Negotiation successful
0925 04:54:24.839921 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":224,"threads_started":1}
I20260925 04:54:24.841213 7297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:24.822568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:46465 (local address 127.6.29.131:58055)
0925 04:54:24.823280 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:24.823327 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:24.825073 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:24.825768 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:24.825777 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:24.826299 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:54:24.826878 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:24.826895 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:24.828042 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:24.828046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:24.829857 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:24.829874 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:24.830677 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:24.830682 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:24.830891 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:24.831653 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:54:24.831675 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:54:24.834404 (+ 2729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:54:24.837379 (+ 2975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:24.837387 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:24.837401 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:24.837806 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:24.838165 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:24.838172 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:24.838179 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:24.838356 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:24.839053 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:54:24.839059 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:54:24.839368 (+ 309us) client_negotiation.cc:769] Sending connection context
0925 04:54:24.839602 (+ 234us) client_negotiation.cc:240] Negotiation successful
0925 04:54:24.840071 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":284,"threads_started":1}
I20260925 04:54:24.842621 7294 heartbeater.cc:347] Connected to a master server at 127.6.29.190:46465
I20260925 04:54:24.842914 7294 heartbeater.cc:464] Registering TS with master...
I20260925 04:54:24.843616 7294 heartbeater.cc:511] Master 127.6.29.190:46465 requested a full tablet report, sending...
I20260925 04:54:24.845148 6837 ts_manager.cc:194] Registered new tserver with Master: 86247c2984bc451badd0a9eabac12a2c (127.6.29.131:37453)
I20260925 04:54:24.845909 6262 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:54:24.846349 6837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.131@KRBTEST.COM'} at 127.6.29.131:58055
I20260925 04:54:24.865074 7296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:24.852239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53182 (local address 127.6.29.190:46465)
0925 04:54:24.852392 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:24.852396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:24.852903 (+ 507us) server_negotiation.cc:407] Connection header received
0925 04:54:24.853110 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:24.853115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:24.853209 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:24.853315 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:24.854812 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:24.855524 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:24.856857 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:24.857271 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:24.857405 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:24.859729 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:24.859735 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:24.859740 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:24.859835 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:24.862373 (+ 2538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:24.863015 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:24.863022 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:24.863025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:24.863152 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:24.863517 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:24.863524 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:24.863528 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:24.863936 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:24.864178 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:24.864516 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 04:54:24.864741 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 04:54:24.871660 6837 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53182:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:54:24.878566 6837 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:54:24.903291 7309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:24.891621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40932 (local address 127.6.29.129:43935)
0925 04:54:24.894990 (+ 3369us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:24.894996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:24.895019 (+ 23us) server_negotiation.cc:407] Connection header received
0925 04:54:24.895101 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:24.895106 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:24.895296 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:24.895514 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:54:24.896157 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:24.897966 (+ 1809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:24.900559 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:24.902134 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:24.902283 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:24.902519 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:24.902794 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 04:54:24.902962 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3211,"thread_start_us":3067,"threads_started":1}
I20260925 04:54:24.903311 7307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:24.891373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:43935 (local address 127.0.0.1:40932)
0925 04:54:24.892428 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:24.892467 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:24.892645 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:24.895504 (+ 2859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:24.895511 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:24.895539 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:54:24.895956 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:24.895971 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:24.898151 (+ 2180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:24.898156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:24.900399 (+ 2243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:24.900415 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:24.902336 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:24.902342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:24.902523 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:24.902568 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 04:54:24.902710 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 04:54:24.902906 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"mutex_wait_us":108,"thread_start_us":1471,"threads_started":1}
I20260925 04:54:24.904454 7308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:24.892223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:38321 (local address 127.0.0.1:55542)
0925 04:54:24.895331 (+ 3108us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:24.895354 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:24.895626 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:24.896547 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:24.896550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:24.896569 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:54:24.896938 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:24.896949 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:24.899027 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:24.899033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:24.901411 (+ 2378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:24.901430 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:24.903444 (+ 2014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:24.903450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:24.903674 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:24.903705 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 04:54:24.903910 (+ 205us) client_negotiation.cc:240] Negotiation successful
0925 04:54:24.904090 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2960,"mutex_wait_us":106,"thread_start_us":1931,"threads_started":1}
I20260925 04:54:24.904481 7312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:24.892717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55542 (local address 127.6.29.130:38321)
0925 04:54:24.893968 (+ 1251us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:24.893981 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:24.895479 (+ 1498us) server_negotiation.cc:407] Connection header received
0925 04:54:24.895893 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:24.895905 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:24.896192 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:24.896325 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:54:24.897158 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:24.898885 (+ 1727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:24.901587 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:24.902958 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:24.903084 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:24.903292 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:24.904027 (+ 735us) server_negotiation.cc:299] Negotiation successful
0925 04:54:24.904187 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1049,"thread_start_us":896,"threads_started":1}
I20260925 04:54:24.904896 7311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:24.892285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45870 (local address 127.6.29.131:37453)
0925 04:54:24.896294 (+ 4009us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:24.896304 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:24.896335 (+ 31us) server_negotiation.cc:407] Connection header received
0925 04:54:24.896427 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:24.896434 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:24.896654 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:24.896850 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:54:24.897516 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:24.899097 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:24.901940 (+ 2843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:24.903356 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:24.904060 (+ 704us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:24.904276 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:24.904342 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 04:54:24.904515 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3826,"thread_start_us":3651,"threads_started":1}
I20260925 04:54:24.905527 7310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:24.891601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.131:37453 (local address 127.0.0.1:45870)
0925 04:54:24.894401 (+ 2800us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:24.894416 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:24.894501 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:24.896858 (+ 2357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:24.896865 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:24.896895 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:54:24.897299 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:24.897312 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:24.899248 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:24.899255 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:24.901792 (+ 2537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:24.901814 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:24.903608 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:24.903614 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:24.903818 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:24.903853 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 04:54:24.903922 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 04:54:24.905204 (+ 1282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2699,"mutex_wait_us":1178,"thread_start_us":3441,"threads_started":1}
I20260925 04:54:24.907828 6961 tablet_service.cc:1467] Processing CreateTablet for tablet 0f19f665abb641608ec8a41e21dc5f4b (DEFAULT_TABLE table=test-table [id=ba321e2621d04f1ba2cb1fd749cc5bbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:54:24.907967 7095 tablet_service.cc:1467] Processing CreateTablet for tablet 0f19f665abb641608ec8a41e21dc5f4b (DEFAULT_TABLE table=test-table [id=ba321e2621d04f1ba2cb1fd749cc5bbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:54:24.909231 6961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f19f665abb641608ec8a41e21dc5f4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:54:24.909451 7095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f19f665abb641608ec8a41e21dc5f4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:54:24.910609 7229 tablet_service.cc:1467] Processing CreateTablet for tablet 0f19f665abb641608ec8a41e21dc5f4b (DEFAULT_TABLE table=test-table [id=ba321e2621d04f1ba2cb1fd749cc5bbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:54:24.912149 7229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f19f665abb641608ec8a41e21dc5f4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:54:24.921499 7313 tablet_bootstrap.cc:492] T 0f19f665abb641608ec8a41e21dc5f4b P 83a6bc29f6f74af7b182fd004cf414d5: Bootstrap starting.
I20260925 04:54:24.923732 7314 tablet_bootstrap.cc:492] T 0f19f665abb641608ec8a41e21dc5f4b P 6a6eb8c2877c4af0b6e7df8e3b2bc8e5: Bootstrap starting.
I20260925 04:54:24.925323 7313 tablet_bootstrap.cc:654] T 0f19f665abb641608ec8a41e21dc5f4b P 83a6bc29f6f74af7b182fd004cf414d5: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:24.926666 7313 log.cc:824] T 0f19f665abb641608ec8a41e21dc5f4b P 83a6bc29f6f74af7b182fd004cf414d5: Log is configured to *not* fsync() on all Append() calls
I20260925 04:54:24.927423 7314 tablet_bootstrap.cc:654] T 0f19f665abb641608ec8a41e21dc5f4b P 6a6eb8c2877c4af0b6e7df8e3b2bc8e5: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:24.928787 7314 log.cc:824] T 0f19f665abb641608ec8a41e21dc5f4b P 6a6eb8c2877c4af0b6e7df8e3b2bc8e5: Log is configured to *not* fsync() on all Append() calls
I20260925 04:54:24.929968 7313 tablet_bootstrap.cc:492] T 0f19f665abb641608ec8a41e21dc5f4b P 83a6bc29f6f74af7b182fd004cf414d5: No bootstrap required, opened a new log
I20260925 04:54:24.930243 7313 ts_tablet_manager.cc:1397] T 0f19f665abb641608ec8a41e21dc5f4b P 83a6bc29f6f74af7b182fd004cf414d5: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260925 04:54:24.930261 7315 tablet_bootstrap.cc:492] T 0f19f665abb641608ec8a41e21dc5f4b P 86247c2984bc451badd0a9eabac12a2c: Bootstrap starting.
I20260925 04:54:24.931620 7314 tablet_bootstrap.cc:492] T 0f19f665abb641608ec8a41e21dc5f4b P 6a6eb8c2877c4af0b6e7df8e3b2bc8e5: No bootstrap required, opened a new log
I20260925 04:54:24.931902 7314 ts_tablet_manager.cc:1397] T 0f19f665abb641608ec8a41e21dc5f4b P 6a6eb8c2877c4af0b6e7df8e3b2bc8e5: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.006s
I20260925 04:54:24.933122 7315 tablet_bootstrap.cc:654] T 0f19f665abb641608ec8a41e21dc5f4b P 86247c2984bc451badd0a9eabac12a2c: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:24.934214 7315 log.cc:824] T 0f19f665abb641608ec8a41e21dc5f4b P 86247c2984bc451badd0a9eabac12a2c: Log is configured to *not* fsync() on all Append() calls
I20260925 04:54:24.936803 7315 tablet_bootstrap.cc:492] T 0f19f665abb641608ec8a41e21dc5f4b P 86247c2984bc451badd0a9eabac12a2c: No bootstrap required, opened a new log
I20260925 04:54:24.937059 7315 ts_tablet_manager.cc:1397] T 0f19f665abb641608ec8a41e21dc5f4b P 86247c2984bc451badd0a9eabac12a2c: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260925 04:54:24.941089 7313 raft_consensus.cc:348] T 0f19f665abb641608ec8a41e21dc5f4b P 83a6bc29f6f74af7b182fd004cf414d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a6bc29f6f74af7b182fd004cf414d5" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 43935 } } peers { permanent_uuid: "86247c2984bc451badd0a9eabac12a2c" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37453 } } peers { permanent_uuid: "6a6eb8c2877c4af0b6e7df8e3b2bc8e5" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 38321 } }
I20260925 04:54:24.941672 7313 raft_consensus.cc:374] T 0f19f665abb641608ec8a41e21dc5f4b P 83a6bc29f6f74af7b182fd004cf414d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:24.941833 7313 raft_consensus.cc:729] T 0f19f665abb641608ec8a41e21dc5f4b P 83a6bc29f6f74af7b182fd004cf414d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83a6bc29f6f74af7b182fd004cf414d5, State: Initialized, Role: FOLLOWER
I20260925 04:54:24.942569 7313 consensus_queue.cc:260] T 0f19f665abb641608ec8a41e21dc5f4b P 83a6bc29f6f74af7b182fd004cf414d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a6bc29f6f74af7b182fd004cf414d5" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 43935 } } peers { permanent_uuid: "86247c2984bc451badd0a9eabac12a2c" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37453 } } peers { permanent_uuid: "6a6eb8c2877c4af0b6e7df8e3b2bc8e5" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 38321 } }
I20260925 04:54:24.942464 7314 raft_consensus.cc:348] T 0f19f665abb641608ec8a41e21dc5f4b P 6a6eb8c2877c4af0b6e7df8e3b2bc8e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a6bc29f6f74af7b182fd004cf414d5" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 43935 } } peers { permanent_uuid: "86247c2984bc451badd0a9eabac12a2c" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37453 } } peers { permanent_uuid: "6a6eb8c2877c4af0b6e7df8e3b2bc8e5" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 38321 } }
I20260925 04:54:24.942888 7314 raft_consensus.cc:374] T 0f19f665abb641608ec8a41e21dc5f4b P 6a6eb8c2877c4af0b6e7df8e3b2bc8e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:24.943032 7314 raft_consensus.cc:729] T 0f19f665abb641608ec8a41e21dc5f4b P 6a6eb8c2877c4af0b6e7df8e3b2bc8e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a6eb8c2877c4af0b6e7df8e3b2bc8e5, State: Initialized, Role: FOLLOWER
I20260925 04:54:24.943795 7314 consensus_queue.cc:260] T 0f19f665abb641608ec8a41e21dc5f4b P 6a6eb8c2877c4af0b6e7df8e3b2bc8e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a6bc29f6f74af7b182fd004cf414d5" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 43935 } } peers { permanent_uuid: "86247c2984bc451badd0a9eabac12a2c" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37453 } } peers { permanent_uuid: "6a6eb8c2877c4af0b6e7df8e3b2bc8e5" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 38321 } }
I20260925 04:54:24.944680 7313 ts_tablet_manager.cc:1428] T 0f19f665abb641608ec8a41e21dc5f4b P 83a6bc29f6f74af7b182fd004cf414d5: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260925 04:54:24.946221 7314 ts_tablet_manager.cc:1428] T 0f19f665abb641608ec8a41e21dc5f4b P 6a6eb8c2877c4af0b6e7df8e3b2bc8e5: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260925 04:54:24.945927 7315 raft_consensus.cc:348] T 0f19f665abb641608ec8a41e21dc5f4b P 86247c2984bc451badd0a9eabac12a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a6bc29f6f74af7b182fd004cf414d5" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 43935 } } peers { permanent_uuid: "86247c2984bc451badd0a9eabac12a2c" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37453 } } peers { permanent_uuid: "6a6eb8c2877c4af0b6e7df8e3b2bc8e5" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 38321 } }
I20260925 04:54:24.946362 7315 raft_consensus.cc:374] T 0f19f665abb641608ec8a41e21dc5f4b P 86247c2984bc451badd0a9eabac12a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:24.946516 7315 raft_consensus.cc:729] T 0f19f665abb641608ec8a41e21dc5f4b P 86247c2984bc451badd0a9eabac12a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86247c2984bc451badd0a9eabac12a2c, State: Initialized, Role: FOLLOWER
I20260925 04:54:24.947351 7315 consensus_queue.cc:260] T 0f19f665abb641608ec8a41e21dc5f4b P 86247c2984bc451badd0a9eabac12a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a6bc29f6f74af7b182fd004cf414d5" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 43935 } } peers { permanent_uuid: "86247c2984bc451badd0a9eabac12a2c" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37453 } } peers { permanent_uuid: "6a6eb8c2877c4af0b6e7df8e3b2bc8e5" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 38321 } }
I20260925 04:54:24.948760 7294 heartbeater.cc:503] Master 127.6.29.190:46465 was elected leader, sending a full tablet report...
I20260925 04:54:24.949641 7315 ts_tablet_manager.cc:1428] T 0f19f665abb641608ec8a41e21dc5f4b P 86247c2984bc451badd0a9eabac12a2c: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
W20260925 04:54:25.003736 7161 tablet.cc:2369] T 0f19f665abb641608ec8a41e21dc5f4b P 6a6eb8c2877c4af0b6e7df8e3b2bc8e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:54:25.030756 7319 raft_consensus.cc:482] T 0f19f665abb641608ec8a41e21dc5f4b P 83a6bc29f6f74af7b182fd004cf414d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:54:25.031090 7319 raft_consensus.cc:504] T 0f19f665abb641608ec8a41e21dc5f4b P 83a6bc29f6f74af7b182fd004cf414d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a6bc29f6f74af7b182fd004cf414d5" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 43935 } } peers { permanent_uuid: "86247c2984bc451badd0a9eabac12a2c" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37453 } } peers { permanent_uuid: "6a6eb8c2877c4af0b6e7df8e3b2bc8e5" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 38321 } }
I20260925 04:54:25.033026 7319 leader_election.cc:288] T 0f19f665abb641608ec8a41e21dc5f4b P 83a6bc29f6f74af7b182fd004cf414d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86247c2984bc451badd0a9eabac12a2c (127.6.29.131:37453), 6a6eb8c2877c4af0b6e7df8e3b2bc8e5 (127.6.29.130:38321)
I20260925 04:54:25.040445 7323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:25.033491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:38321 (local address 127.6.29.129:46347)
0925 04:54:25.034509 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:25.034553 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:25.034683 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:25.035029 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:25.035036 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:25.035064 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:54:25.035451 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:25.035462 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:25.036567 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:25.036571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:25.038255 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:25.038268 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:25.039638 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:25.039644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:25.039880 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:25.039929 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 04:54:25.040031 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 04:54:25.040142 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":902,"mutex_wait_us":116,"spinlock_wait_cycles":23808,"thread_start_us":647,"threads_started":1}
I20260925 04:54:25.040433 7311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:25.033426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:54757 (local address 127.6.29.131:37453)
0925 04:54:25.033635 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:25.033642 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:25.034461 (+ 819us) server_negotiation.cc:407] Connection header received
0925 04:54:25.034639 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:25.034645 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:25.034738 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:25.034832 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:54:25.035461 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:25.036377 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:25.038438 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:25.039320 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:25.039470 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:25.039563 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:25.040054 (+ 491us) server_negotiation.cc:299] Negotiation successful
0925 04:54:25.040192 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 04:54:25.040521 7312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:25.033653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:46347 (local address 127.6.29.130:38321)
0925 04:54:25.033889 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:25.033898 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:25.034648 (+ 750us) server_negotiation.cc:407] Connection header received
0925 04:54:25.034819 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:25.034827 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:25.034903 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:25.035002 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:54:25.035606 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:25.036408 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:25.038410 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:25.039368 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:25.039554 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:25.039627 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:25.040174 (+ 547us) server_negotiation.cc:299] Negotiation successful
0925 04:54:25.040298 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 04:54:25.041147 7322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:25.033451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.131:37453 (local address 127.6.29.129:54757)
0925 04:54:25.034330 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:25.034346 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:25.034487 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:25.034869 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:25.034874 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:25.034897 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:54:25.035298 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:25.035314 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:25.036508 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:25.036515 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:25.038264 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:25.038276 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:25.039536 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:25.039544 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:25.039751 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:25.039869 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 04:54:25.040664 (+ 795us) client_negotiation.cc:240] Negotiation successful
0925 04:54:25.040806 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"mutex_wait_us":6,"spinlock_wait_cycles":29056,"thread_start_us":468,"threads_started":1}
I20260925 04:54:25.041939 7115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f19f665abb641608ec8a41e21dc5f4b" candidate_uuid: "83a6bc29f6f74af7b182fd004cf414d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6eb8c2877c4af0b6e7df8e3b2bc8e5" is_pre_election: true
I20260925 04:54:25.042032 7249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f19f665abb641608ec8a41e21dc5f4b" candidate_uuid: "83a6bc29f6f74af7b182fd004cf414d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86247c2984bc451badd0a9eabac12a2c" is_pre_election: true
I20260925 04:54:25.042493 7115 raft_consensus.cc:2454] T 0f19f665abb641608ec8a41e21dc5f4b P 6a6eb8c2877c4af0b6e7df8e3b2bc8e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83a6bc29f6f74af7b182fd004cf414d5 in term 0.
I20260925 04:54:25.042542 7249 raft_consensus.cc:2454] T 0f19f665abb641608ec8a41e21dc5f4b P 86247c2984bc451badd0a9eabac12a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83a6bc29f6f74af7b182fd004cf414d5 in term 0.
I20260925 04:54:25.043260 6915 leader_election.cc:302] T 0f19f665abb641608ec8a41e21dc5f4b P 83a6bc29f6f74af7b182fd004cf414d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6a6eb8c2877c4af0b6e7df8e3b2bc8e5, 83a6bc29f6f74af7b182fd004cf414d5, 86247c2984bc451badd0a9eabac12a2c; no voters:
I20260925 04:54:25.043674 7319 raft_consensus.cc:2781] T 0f19f665abb641608ec8a41e21dc5f4b P 83a6bc29f6f74af7b182fd004cf414d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:54:25.043919 7319 raft_consensus.cc:482] T 0f19f665abb641608ec8a41e21dc5f4b P 83a6bc29f6f74af7b182fd004cf414d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:54:25.043999 7319 raft_consensus.cc:3037] T 0f19f665abb641608ec8a41e21dc5f4b P 83a6bc29f6f74af7b182fd004cf414d5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:25.046181 7319 raft_consensus.cc:504] T 0f19f665abb641608ec8a41e21dc5f4b P 83a6bc29f6f74af7b182fd004cf414d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a6bc29f6f74af7b182fd004cf414d5" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 43935 } } peers { permanent_uuid: "86247c2984bc451badd0a9eabac12a2c" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37453 } } peers { permanent_uuid: "6a6eb8c2877c4af0b6e7df8e3b2bc8e5" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 38321 } }
I20260925 04:54:25.046758 7319 leader_election.cc:288] T 0f19f665abb641608ec8a41e21dc5f4b P 83a6bc29f6f74af7b182fd004cf414d5 [CANDIDATE]: Term 1 election: Requested vote from peers 86247c2984bc451badd0a9eabac12a2c (127.6.29.131:37453), 6a6eb8c2877c4af0b6e7df8e3b2bc8e5 (127.6.29.130:38321)
I20260925 04:54:25.047242 7249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f19f665abb641608ec8a41e21dc5f4b" candidate_uuid: "83a6bc29f6f74af7b182fd004cf414d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86247c2984bc451badd0a9eabac12a2c"
I20260925 04:54:25.047452 7249 raft_consensus.cc:3037] T 0f19f665abb641608ec8a41e21dc5f4b P 86247c2984bc451badd0a9eabac12a2c [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:25.047647 7115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f19f665abb641608ec8a41e21dc5f4b" candidate_uuid: "83a6bc29f6f74af7b182fd004cf414d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6eb8c2877c4af0b6e7df8e3b2bc8e5"
I20260925 04:54:25.047890 7115 raft_consensus.cc:3037] T 0f19f665abb641608ec8a41e21dc5f4b P 6a6eb8c2877c4af0b6e7df8e3b2bc8e5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:25.050920 7249 raft_consensus.cc:2454] T 0f19f665abb641608ec8a41e21dc5f4b P 86247c2984bc451badd0a9eabac12a2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83a6bc29f6f74af7b182fd004cf414d5 in term 1.
I20260925 04:54:25.051039 7115 raft_consensus.cc:2454] T 0f19f665abb641608ec8a41e21dc5f4b P 6a6eb8c2877c4af0b6e7df8e3b2bc8e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83a6bc29f6f74af7b182fd004cf414d5 in term 1.
I20260925 04:54:25.051396 6915 leader_election.cc:302] T 0f19f665abb641608ec8a41e21dc5f4b P 83a6bc29f6f74af7b182fd004cf414d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83a6bc29f6f74af7b182fd004cf414d5, 86247c2984bc451badd0a9eabac12a2c; no voters:
I20260925 04:54:25.051707 7319 raft_consensus.cc:2781] T 0f19f665abb641608ec8a41e21dc5f4b P 83a6bc29f6f74af7b182fd004cf414d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:54:25.052206 7319 raft_consensus.cc:686] T 0f19f665abb641608ec8a41e21dc5f4b P 83a6bc29f6f74af7b182fd004cf414d5 [term 1 LEADER]: Becoming Leader. State: Replica: 83a6bc29f6f74af7b182fd004cf414d5, State: Running, Role: LEADER
I20260925 04:54:25.052680 7319 consensus_queue.cc:237] T 0f19f665abb641608ec8a41e21dc5f4b P 83a6bc29f6f74af7b182fd004cf414d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a6bc29f6f74af7b182fd004cf414d5" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 43935 } } peers { permanent_uuid: "86247c2984bc451badd0a9eabac12a2c" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37453 } } peers { permanent_uuid: "6a6eb8c2877c4af0b6e7df8e3b2bc8e5" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 38321 } }
I20260925 04:54:25.057273 6835 catalog_manager.cc:5495] T 0f19f665abb641608ec8a41e21dc5f4b P 83a6bc29f6f74af7b182fd004cf414d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83a6bc29f6f74af7b182fd004cf414d5 (127.6.29.129). New cstate: current_term: 1 leader_uuid: "83a6bc29f6f74af7b182fd004cf414d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83a6bc29f6f74af7b182fd004cf414d5" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 43935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86247c2984bc451badd0a9eabac12a2c" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a6eb8c2877c4af0b6e7df8e3b2bc8e5" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 38321 } health_report { overall_health: UNKNOWN } } }
W20260925 04:54:25.071009 7295 tablet.cc:2369] T 0f19f665abb641608ec8a41e21dc5f4b P 86247c2984bc451badd0a9eabac12a2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 04:54:25 dist-test-slave-xzjr krb5kdc[6792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.129@KRBTEST.COM
I20260925 04:54:25.111429 7309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:25.095738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40938 (local address 127.6.29.129:43935)
0925 04:54:25.096028 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:25.096034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:25.096655 (+ 621us) server_negotiation.cc:407] Connection header received
0925 04:54:25.096897 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:25.096904 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:25.097006 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:25.097127 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:25.098661 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:25.099397 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:25.100839 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:25.101259 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:25.101398 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:25.105359 (+ 3961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:25.105375 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:25.105393 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:25.105426 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:25.108428 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:25.109178 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:25.109185 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:25.109187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:25.109328 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:25.109747 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:25.109750 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:25.109753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:25.110090 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:25.110391 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:25.110690 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 04:54:25.110949 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
Sep 25 04:54:25 dist-test-slave-xzjr krb5kdc[6792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312065, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 04:54:25 dist-test-slave-xzjr krb5kdc[6792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312065, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.29.129@KRBTEST.COM
I20260925 04:54:25.159220 7309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:25.145112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40952 (local address 127.6.29.129:43935)
0925 04:54:25.145334 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:25.145343 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:25.145725 (+ 382us) server_negotiation.cc:407] Connection header received
0925 04:54:25.145961 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:25.145965 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:25.146030 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:25.146139 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:25.147573 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:25.148441 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:25.149705 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:25.150139 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:25.150269 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:25.154028 (+ 3759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:25.154033 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:25.154037 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:25.154068 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:25.156639 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:25.157328 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:25.157333 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:25.157337 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:25.157468 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:25.157850 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:25.157857 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:25.157860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:25.158102 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:25.158386 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:25.158652 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 04:54:25.158890 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260925 04:54:25.173679 7027 tablet.cc:2369] T 0f19f665abb641608ec8a41e21dc5f4b P 83a6bc29f6f74af7b182fd004cf414d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 04:54:25 dist-test-slave-xzjr krb5kdc[6792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 04:54:25 dist-test-slave-xzjr krb5kdc[6792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.129@KRBTEST.COM
I20260925 04:54:25.205163 7309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:25.191526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40960 (local address 127.6.29.129:43935)
0925 04:54:25.191823 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:25.191830 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:25.192070 (+ 240us) server_negotiation.cc:407] Connection header received
0925 04:54:25.192233 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:25.192237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:25.192316 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:25.192449 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:25.193745 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:25.194546 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:25.195824 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:25.196224 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:25.196358 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:25.200326 (+ 3968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:25.200331 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:25.200335 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:25.200370 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:25.202880 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:25.203474 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:25.203478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:25.203480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:25.203629 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:25.204043 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:25.204047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:25.204049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:25.204322 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:25.204521 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:25.204733 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 04:54:25.204901 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
W20260925 04:54:25.207005 6941 tablet_service.cc:2908] Rejecting scan request for tablet 0f19f665abb641608ec8a41e21dc5f4b: Uninitialized: clean time has not yet been initialized
I20260925 04:54:25.211009 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 6898
I20260925 04:54:25.223134 7309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:25.211860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40970 (local address 127.6.29.129:43935)
0925 04:54:25.212066 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:25.212076 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:25.212341 (+ 265us) server_negotiation.cc:407] Connection header received
0925 04:54:25.212453 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:25.212456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:25.212515 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:25.212591 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:25.213681 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:25.214524 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:25.215659 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:25.216125 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:25.216218 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:25.218331 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:25.218335 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:25.218338 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:25.218369 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:25.220903 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:25.221486 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:25.221490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:25.221492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:25.221604 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:25.221915 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:25.221918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:25.221920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:25.222172 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:25.222347 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:25.222584 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 04:54:25.222778 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 04:54:25.223855 7021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:54:25.317625 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 6898
I20260925 04:54:25.335166 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 7032
Sep 25 04:54:25 dist-test-slave-xzjr krb5kdc[6792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.130@KRBTEST.COM
I20260925 04:54:25.349628 7312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:25.336057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55558 (local address 127.6.29.130:38321)
0925 04:54:25.336217 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:25.336223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:25.336241 (+ 18us) server_negotiation.cc:407] Connection header received
0925 04:54:25.336296 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:25.336299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:25.336379 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:25.336462 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:25.337586 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:25.338464 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:25.339845 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:25.340266 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:25.340359 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:25.344195 (+ 3836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:25.344209 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:25.344226 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:25.344259 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:25.347033 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:25.347640 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:25.347648 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:25.347650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:25.347958 (+ 308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:25.348288 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:25.348292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:25.348295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:25.348666 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:25.348936 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:25.349084 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 04:54:25.349234 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 04:54:25.350335 7155 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:54:25.439159 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 7032
I20260925 04:54:25.456051 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 7166
Sep 25 04:54:25 dist-test-slave-xzjr krb5kdc[6792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.131@KRBTEST.COM
I20260925 04:54:25.471314 7311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:25.456893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45884 (local address 127.6.29.131:37453)
0925 04:54:25.457119 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:25.457126 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:25.457150 (+ 24us) server_negotiation.cc:407] Connection header received
0925 04:54:25.457227 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:25.457231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:25.457294 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:25.457387 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:25.458667 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:25.459409 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:25.460770 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:25.461178 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:25.461279 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:25.465384 (+ 4105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:25.465408 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:25.465427 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:25.465468 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:25.468560 (+ 3092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:25.469407 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:25.469415 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:25.469417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:25.469549 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:25.469931 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:25.469934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:25.469936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:25.470302 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:25.470536 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:25.470848 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 04:54:25.471012 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 04:54:25.472245 7289 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:54:25.561993 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 7166
I20260925 04:54:25.578356 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 6805
I20260925 04:54:25.579562 6867 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:54:25.687682 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 6805
2026-09-25T04:54:25Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (7956 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:54:25 dist-test-slave-xzjr krb5kdc[7343](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:54:25 dist-test-slave-xzjr krb5kdc[7343](info): set up 2 sockets
Sep 25 04:54:25 dist-test-slave-xzjr krb5kdc[7343](info): commencing operation
krb5kdc: starting...
W20260925 04:54:26.796452 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.007s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 04:54:26 dist-test-slave-xzjr krb5kdc[7343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:54:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:54:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.29.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
I20260925 04:54:26.984192 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:33387
--webserver_interface=127.6.29.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:44197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:33387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:54:27.311012 7356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:54:27.311358 7356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:54:27.311522 7356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:54:27.318727 7356 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:54:27.318881 7356 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 04:54:27.318977 7356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:54:27.319067 7356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:54:27.319157 7356 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:54:27.319243 7356 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:54:27.329150 7356 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:44197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:33387
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:33387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:54:27.330475 7356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:54:27.331845 7356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:54:27.344087 7364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:27.344087 7362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:27.344663 7361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:54:27.345099 7356 server_base.cc:1034] running on GCE node
I20260925 04:54:27.345884 7356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:54:27.347313 7356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:54:27.348546 7356 hybrid_clock.cc:648] HybridClock initialized: now 1790312067348456 us; error 56 us; skew 500 ppm
I20260925 04:54:27.348932 7356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:54:27 dist-test-slave-xzjr krb5kdc[7343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312067, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:54:27.352216 7356 init.cc:377] Logged in from keytab as kudu/127.6.29.190@KRBTEST.COM (short username kudu)
I20260925 04:54:27.353696 7356 webserver.cc:460] Webserver started at http://127.6.29.190:34779/ using document root <none> and password file <none>
I20260925 04:54:27.354384 7356 fs_manager.cc:359] Metadata directory not provided
I20260925 04:54:27.354470 7356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:54:27.354744 7356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:54:27.357704 7356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "b60b88bcac904dd584ad8b377a235dc1"
format_stamp: "Formatted at 2026-09-25 04:54:27 on dist-test-slave-xzjr"
server_key: "a39669f00742a99e98cf1ebd23c92d10"
server_key_iv: "8518e40295c779007b23cad7d697ae9b"
server_key_version: "encryptionkey@0"
I20260925 04:54:27.358449 7356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "b60b88bcac904dd584ad8b377a235dc1"
format_stamp: "Formatted at 2026-09-25 04:54:27 on dist-test-slave-xzjr"
server_key: "a39669f00742a99e98cf1ebd23c92d10"
server_key_iv: "8518e40295c779007b23cad7d697ae9b"
server_key_version: "encryptionkey@0"
I20260925 04:54:27.363601 7356 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 04:54:27.367046 7371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:27.368445 7356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:54:27.368624 7356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "b60b88bcac904dd584ad8b377a235dc1"
format_stamp: "Formatted at 2026-09-25 04:54:27 on dist-test-slave-xzjr"
server_key: "a39669f00742a99e98cf1ebd23c92d10"
server_key_iv: "8518e40295c779007b23cad7d697ae9b"
server_key_version: "encryptionkey@0"
I20260925 04:54:27.368846 7356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:54:27.383440 7356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:54:27.387979 7356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:54:27.388265 7356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:54:27.410524 7356 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:33387
I20260925 04:54:27.410542 7432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:33387 every 8 connection(s)
I20260925 04:54:27.412102 7356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:54:27.416283 7433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:54:27.422464 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 7356
I20260925 04:54:27.422689 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 04:54:27.423357 6262 external_mini_cluster.cc:1428] Setting key 89bc43da2d6883b4b2e5349709e3073a
I20260925 04:54:27.427079 7433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b60b88bcac904dd584ad8b377a235dc1: Bootstrap starting.
I20260925 04:54:27.431659 7433 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b60b88bcac904dd584ad8b377a235dc1: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:27.432834 7433 log.cc:824] T 00000000000000000000000000000000 P b60b88bcac904dd584ad8b377a235dc1: Log is configured to *not* fsync() on all Append() calls
I20260925 04:54:27.436285 7433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b60b88bcac904dd584ad8b377a235dc1: No bootstrap required, opened a new log
Sep 25 04:54:27 dist-test-slave-xzjr krb5kdc[7343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:54:27.446386 7433 raft_consensus.cc:348] T 00000000000000000000000000000000 P b60b88bcac904dd584ad8b377a235dc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b60b88bcac904dd584ad8b377a235dc1" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33387 } }
I20260925 04:54:27.446853 7433 raft_consensus.cc:374] T 00000000000000000000000000000000 P b60b88bcac904dd584ad8b377a235dc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:27.446970 7433 raft_consensus.cc:729] T 00000000000000000000000000000000 P b60b88bcac904dd584ad8b377a235dc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b60b88bcac904dd584ad8b377a235dc1, State: Initialized, Role: FOLLOWER
I20260925 04:54:27.447890 7433 consensus_queue.cc:260] T 00000000000000000000000000000000 P b60b88bcac904dd584ad8b377a235dc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b60b88bcac904dd584ad8b377a235dc1" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33387 } }
I20260925 04:54:27.448176 7433 raft_consensus.cc:388] T 00000000000000000000000000000000 P b60b88bcac904dd584ad8b377a235dc1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:54:27.448318 7433 raft_consensus.cc:482] T 00000000000000000000000000000000 P b60b88bcac904dd584ad8b377a235dc1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:54:27.448519 7433 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b60b88bcac904dd584ad8b377a235dc1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:27.451021 7433 raft_consensus.cc:504] T 00000000000000000000000000000000 P b60b88bcac904dd584ad8b377a235dc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b60b88bcac904dd584ad8b377a235dc1" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33387 } }
I20260925 04:54:27.451411 7436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:27.425753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53160 (local address 127.6.29.190:33387)
0925 04:54:27.426808 (+ 1055us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:27.426825 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:27.427267 (+ 442us) server_negotiation.cc:407] Connection header received
0925 04:54:27.430099 (+ 2832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:27.430131 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:27.430725 (+ 594us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:27.431347 (+ 622us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:27.432824 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:27.434715 (+ 1891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:27.436162 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:27.437304 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:27.437493 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:27.442026 (+ 4533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:27.442037 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:27.442061 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:27.442127 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:27.446329 (+ 4202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:27.447275 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:27.447284 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:27.447293 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:27.447536 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:27.448183 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:27.448187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:27.448189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:27.448926 (+ 737us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:27.449313 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:27.449771 (+ 458us) server_negotiation.cc:299] Negotiation successful
0925 04:54:27.450233 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":769,"thread_start_us":480,"threads_started":1}
I20260925 04:54:27.451653 7433 leader_election.cc:302] T 00000000000000000000000000000000 P b60b88bcac904dd584ad8b377a235dc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b60b88bcac904dd584ad8b377a235dc1; no voters:
I20260925 04:54:27.452376 7433 leader_election.cc:288] T 00000000000000000000000000000000 P b60b88bcac904dd584ad8b377a235dc1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:54:27.452502 7438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b60b88bcac904dd584ad8b377a235dc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:54:27.453204 7438 raft_consensus.cc:686] T 00000000000000000000000000000000 P b60b88bcac904dd584ad8b377a235dc1 [term 1 LEADER]: Becoming Leader. State: Replica: b60b88bcac904dd584ad8b377a235dc1, State: Running, Role: LEADER
I20260925 04:54:27.453915 7438 consensus_queue.cc:237] T 00000000000000000000000000000000 P b60b88bcac904dd584ad8b377a235dc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b60b88bcac904dd584ad8b377a235dc1" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33387 } }
I20260925 04:54:27.456403 7439 sys_catalog.cc:455] T 00000000000000000000000000000000 P b60b88bcac904dd584ad8b377a235dc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b60b88bcac904dd584ad8b377a235dc1. Latest consensus state: current_term: 1 leader_uuid: "b60b88bcac904dd584ad8b377a235dc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b60b88bcac904dd584ad8b377a235dc1" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33387 } } }
I20260925 04:54:27.456851 7439 sys_catalog.cc:458] T 00000000000000000000000000000000 P b60b88bcac904dd584ad8b377a235dc1 [sys.catalog]: This master's current role is: LEADER
I20260925 04:54:27.457136 7433 sys_catalog.cc:564] T 00000000000000000000000000000000 P b60b88bcac904dd584ad8b377a235dc1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:54:27.457345 7440 sys_catalog.cc:455] T 00000000000000000000000000000000 P b60b88bcac904dd584ad8b377a235dc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b60b88bcac904dd584ad8b377a235dc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b60b88bcac904dd584ad8b377a235dc1" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33387 } } }
I20260925 04:54:27.457746 7440 sys_catalog.cc:458] T 00000000000000000000000000000000 P b60b88bcac904dd584ad8b377a235dc1 [sys.catalog]: This master's current role is: LEADER
I20260925 04:54:27.460776 7442 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:54:27.468874 7442 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:54:27.479645 7442 catalog_manager.cc:1347] Generated new cluster ID: a21ca12a7e1f4b57a4754072e6057c13
I20260925 04:54:27.479813 7442 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:54:27.490267 7442 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:54:27.492170 7442 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:54:27.501799 7442 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b60b88bcac904dd584ad8b377a235dc1: Generated new TSK 0
I20260925 04:54:27.502871 7442 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.29.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab.
Entry for principal kudu/127.6.29.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab.
I20260925 04:54:27.555292 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.129:0
--local_ip_for_outbound_sockets=127.6.29.129
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:33387
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:44197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:54:27.900247 7459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:54:27.900539 7459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:54:27.900636 7459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:54:27.907697 7459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:54:27.907827 7459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:54:27.907910 7459 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 04:54:27.907990 7459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.129
I20260925 04:54:27.917645 7459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:44197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.29.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.29.190:33387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.129
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:54:27.918936 7459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:54:27.920280 7459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:54:27.933460 7464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:27.933501 7467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:54:27.933961 7459 server_base.cc:1034] running on GCE node
W20260925 04:54:27.933540 7465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:54:27.934693 7459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:54:27.935698 7459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:54:27.936952 7459 hybrid_clock.cc:648] HybridClock initialized: now 1790312067936883 us; error 58 us; skew 500 ppm
I20260925 04:54:27.937248 7459 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:54:27 dist-test-slave-xzjr krb5kdc[7343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312067, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:54:27.940339 7459 init.cc:377] Logged in from keytab as kudu/127.6.29.129@KRBTEST.COM (short username kudu)
I20260925 04:54:27.941706 7459 webserver.cc:460] Webserver started at http://127.6.29.129:46031/ using document root <none> and password file <none>
I20260925 04:54:27.942384 7459 fs_manager.cc:359] Metadata directory not provided
I20260925 04:54:27.942510 7459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:54:27.942821 7459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:54:27.945673 7459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-0/data/instance:
uuid: "cc3aa37d80bd4be88566dc19585eef45"
format_stamp: "Formatted at 2026-09-25 04:54:27 on dist-test-slave-xzjr"
server_key: "5c546621c21cfe2d463e87c48e2ebd82"
server_key_iv: "f0dd0c9f4ff5deab3a31051ed40c0723"
server_key_version: "encryptionkey@0"
I20260925 04:54:27.946436 7459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance:
uuid: "cc3aa37d80bd4be88566dc19585eef45"
format_stamp: "Formatted at 2026-09-25 04:54:27 on dist-test-slave-xzjr"
server_key: "5c546621c21cfe2d463e87c48e2ebd82"
server_key_iv: "f0dd0c9f4ff5deab3a31051ed40c0723"
server_key_version: "encryptionkey@0"
I20260925 04:54:27.951619 7459 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260925 04:54:27.955154 7474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:27.956562 7459 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:54:27.956737 7459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-0/wal
uuid: "cc3aa37d80bd4be88566dc19585eef45"
format_stamp: "Formatted at 2026-09-25 04:54:27 on dist-test-slave-xzjr"
server_key: "5c546621c21cfe2d463e87c48e2ebd82"
server_key_iv: "f0dd0c9f4ff5deab3a31051ed40c0723"
server_key_version: "encryptionkey@0"
I20260925 04:54:27.957002 7459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:54:27.975912 7459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:54:27.981220 7459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:54:27.981583 7459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:54:27.984377 7459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:54:27.984508 7459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:27.984665 7459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:54:27.984747 7459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 04:54:27 dist-test-slave-xzjr krb5kdc[7343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312067, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:54:28.009992 7483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:27.985372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:45267 (local address 127.6.29.190:33387)
0925 04:54:27.985859 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:27.985866 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:27.987191 (+ 1325us) server_negotiation.cc:407] Connection header received
0925 04:54:27.989567 (+ 2376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:27.989580 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:27.989677 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:27.989759 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:27.992782 (+ 3023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:27.993834 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:27.996767 (+ 2933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:27.997555 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:27.997681 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:28.002518 (+ 4837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:28.002527 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:28.002534 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:28.002586 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:28.006174 (+ 3588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:28.007366 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:28.007373 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:28.007380 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:28.007535 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:28.008120 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:28.008125 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:28.008129 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:28.008432 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:28.008590 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:28.009390 (+ 800us) server_negotiation.cc:299] Negotiation successful
0925 04:54:28.009629 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":241,"threads_started":1}
I20260925 04:54:28.011385 7484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:27.986085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:33387 (local address 127.6.29.129:45267)
0925 04:54:27.986991 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:27.987047 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:27.989157 (+ 2110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:27.990388 (+ 1231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:27.990396 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:27.991265 (+ 869us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:54:27.992485 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:27.992509 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:27.994069 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:27.994073 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:27.996571 (+ 2498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:27.996589 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:27.998084 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:27.998094 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:27.998427 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:27.999288 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:54:27.999318 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:54:28.002183 (+ 2865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:54:28.006355 (+ 4172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:28.006365 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:28.006388 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:28.007149 (+ 761us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:28.007660 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:28.007665 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:28.007669 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:28.007982 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:28.008587 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:54:28.008596 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:54:28.009006 (+ 410us) client_negotiation.cc:769] Sending connection context
0925 04:54:28.009286 (+ 280us) client_negotiation.cc:240] Negotiation successful
0925 04:54:28.009872 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":411,"threads_started":1}
I20260925 04:54:28.036936 7547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:28.018153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:33387 (local address 127.6.29.129:59421)
0925 04:54:28.019501 (+ 1348us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:28.019532 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:28.019734 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:28.020169 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:28.020173 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:28.021051 (+ 878us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:54:28.021754 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:28.021774 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:28.023725 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:28.023734 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:28.026150 (+ 2416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:28.026170 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:28.027171 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:28.027182 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:28.027494 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:28.028270 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:54:28.028297 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:54:28.029547 (+ 1250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:54:28.033356 (+ 3809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:28.033363 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:28.033367 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:28.034059 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:28.034579 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:28.034584 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:28.034588 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:28.034771 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:28.035506 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:54:28.035513 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:54:28.035758 (+ 245us) client_negotiation.cc:769] Sending connection context
0925 04:54:28.035976 (+ 218us) client_negotiation.cc:240] Negotiation successful
0925 04:54:28.036439 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1140,"thread_start_us":976,"threads_started":1}
I20260925 04:54:28.037097 7483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:28.018110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:59421 (local address 127.6.29.190:33387)
0925 04:54:28.018421 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:28.018430 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:28.019659 (+ 1229us) server_negotiation.cc:407] Connection header received
0925 04:54:28.019936 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:28.019942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:28.020032 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:28.020143 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:28.022005 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:28.023480 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:28.026381 (+ 2901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:28.026904 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:28.027059 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:28.029789 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:28.029798 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:28.029804 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:28.029849 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:28.033065 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:28.034237 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:28.034242 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:28.034245 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:28.034427 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:28.034935 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:28.034944 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:28.034950 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:28.035370 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:28.035555 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:28.036518 (+ 963us) server_negotiation.cc:299] Negotiation successful
0925 04:54:28.036740 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
I20260925 04:54:28.042187 7459 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.129:41377
I20260925 04:54:28.042209 7594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.129:41377 every 8 connection(s)
I20260925 04:54:28.043412 7459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
I20260925 04:54:28.043839 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 7459
I20260925 04:54:28.044085 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance
I20260925 04:54:28.044938 6262 external_mini_cluster.cc:1428] Setting key 767e4c0be836d4076c14adeea40497a8
I20260925 04:54:28.046027 7595 heartbeater.cc:347] Connected to a master server at 127.6.29.190:33387
I20260925 04:54:28.046693 7595 heartbeater.cc:464] Registering TS with master...
I20260925 04:54:28.047485 7595 heartbeater.cc:511] Master 127.6.29.190:33387 requested a full tablet report, sending...
I20260925 04:54:28.049845 7388 ts_manager.cc:194] Registered new tserver with Master: cc3aa37d80bd4be88566dc19585eef45 (127.6.29.129:41377)
I20260925 04:54:28.052049 7388 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.129@KRBTEST.COM'} at 127.6.29.129:45267
WARNING: no policy specified for kudu/127.6.29.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab.
Entry for principal kudu/127.6.29.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab.
I20260925 04:54:28.088189 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.130:0
--local_ip_for_outbound_sockets=127.6.29.130
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:33387
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:44197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:54:28.406937 7599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:54:28.407260 7599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:54:28.407434 7599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:54:28.417604 7599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:54:28.417755 7599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:54:28.417881 7599 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 04:54:28.418004 7599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.130
I20260925 04:54:28.433586 7599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:44197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.29.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.29.190:33387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.130
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:54:28.434947 7599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:54:28.436439 7599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:54:28.449507 7604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:28.449510 7605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:28.450011 7607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:54:28.450371 7599 server_base.cc:1034] running on GCE node
I20260925 04:54:28.450960 7599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:54:28.451977 7599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:54:28.453241 7599 hybrid_clock.cc:648] HybridClock initialized: now 1790312068453155 us; error 58 us; skew 500 ppm
I20260925 04:54:28.453558 7599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:54:28 dist-test-slave-xzjr krb5kdc[7343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312068, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:54:28.456946 7599 init.cc:377] Logged in from keytab as kudu/127.6.29.130@KRBTEST.COM (short username kudu)
I20260925 04:54:28.458616 7599 webserver.cc:460] Webserver started at http://127.6.29.130:35987/ using document root <none> and password file <none>
I20260925 04:54:28.459391 7599 fs_manager.cc:359] Metadata directory not provided
I20260925 04:54:28.459528 7599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:54:28.459929 7599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:54:28.462985 7599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-1/data/instance:
uuid: "f1a6d1bf47274f4288760bb24db5d030"
format_stamp: "Formatted at 2026-09-25 04:54:28 on dist-test-slave-xzjr"
server_key: "c9f3abb9984d96342587419c0694c49d"
server_key_iv: "b35ffea3dcfc3559012efa4fa88d57b3"
server_key_version: "encryptionkey@0"
I20260925 04:54:28.463754 7599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance:
uuid: "f1a6d1bf47274f4288760bb24db5d030"
format_stamp: "Formatted at 2026-09-25 04:54:28 on dist-test-slave-xzjr"
server_key: "c9f3abb9984d96342587419c0694c49d"
server_key_iv: "b35ffea3dcfc3559012efa4fa88d57b3"
server_key_version: "encryptionkey@0"
I20260925 04:54:28.469118 7599 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 04:54:28.472872 7614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:28.474449 7599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:54:28.474591 7599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-1/wal
uuid: "f1a6d1bf47274f4288760bb24db5d030"
format_stamp: "Formatted at 2026-09-25 04:54:28 on dist-test-slave-xzjr"
server_key: "c9f3abb9984d96342587419c0694c49d"
server_key_iv: "b35ffea3dcfc3559012efa4fa88d57b3"
server_key_version: "encryptionkey@0"
I20260925 04:54:28.474774 7599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:54:28.490700 7599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:54:28.495184 7599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:54:28.495406 7599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:54:28.498108 7599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:54:28.498208 7599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:28.498373 7599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:54:28.498441 7599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 04:54:28 dist-test-slave-xzjr krb5kdc[7343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312068, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:54:28.523204 7483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:28.499369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:44761 (local address 127.6.29.190:33387)
0925 04:54:28.499602 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:28.499609 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:28.501144 (+ 1535us) server_negotiation.cc:407] Connection header received
0925 04:54:28.504189 (+ 3045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:28.504198 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:28.504299 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:28.504429 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:28.507206 (+ 2777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:28.508172 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:28.511402 (+ 3230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:28.511919 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:28.512061 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:28.516682 (+ 4621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:28.516687 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:28.516690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:28.516723 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:28.519852 (+ 3129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:28.520850 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:28.520858 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:28.520863 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:28.521000 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:28.521425 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:28.521433 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:28.521438 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:28.521722 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:28.521891 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:28.522626 (+ 735us) server_negotiation.cc:299] Negotiation successful
0925 04:54:28.522841 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 04:54:28.524706 7623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:28.499728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:33387 (local address 127.6.29.130:44761)
0925 04:54:28.500823 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:28.500877 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:28.503800 (+ 2923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:28.504806 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:28.504823 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:28.505812 (+ 989us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:54:28.506970 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:28.507004 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:28.508349 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:28.508358 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:28.511245 (+ 2887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:28.511266 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:28.512158 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:28.512167 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:28.512555 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:28.513660 (+ 1105us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:54:28.513683 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:54:28.516407 (+ 2724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:54:28.520027 (+ 3620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:28.520038 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:28.520052 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:28.520722 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:28.521126 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:28.521129 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:28.521132 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:28.521311 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:28.521868 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:54:28.521877 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:54:28.522249 (+ 372us) client_negotiation.cc:769] Sending connection context
0925 04:54:28.522551 (+ 302us) client_negotiation.cc:240] Negotiation successful
0925 04:54:28.523092 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":347,"threads_started":1}
I20260925 04:54:28.548118 7599 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.130:44993
I20260925 04:54:28.548136 7733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.130:44993 every 8 connection(s)
I20260925 04:54:28.549829 7599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
I20260925 04:54:28.551492 7707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:28.533421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:33387 (local address 127.6.29.130:56027)
0925 04:54:28.536615 (+ 3194us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:28.536637 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:28.536846 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:28.537292 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:28.537296 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:28.537814 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:54:28.538307 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:28.538321 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:28.540000 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:28.540006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:28.541042 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:28.541053 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:28.542064 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:28.542074 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:28.542301 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:28.543114 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:54:28.543150 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:54:28.544016 (+ 866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:54:28.548317 (+ 4301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:28.548323 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:28.548327 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:28.548860 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:28.549327 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:28.549330 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:28.549332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:28.549463 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:28.550164 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:54:28.550167 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:54:28.550363 (+ 196us) client_negotiation.cc:769] Sending connection context
0925 04:54:28.550583 (+ 220us) client_negotiation.cc:240] Negotiation successful
0925 04:54:28.550923 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3045,"thread_start_us":2900,"threads_started":1}
I20260925 04:54:28.552528 7483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:28.533425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:56027 (local address 127.6.29.190:33387)
0925 04:54:28.533658 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:28.533666 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:28.536769 (+ 3103us) server_negotiation.cc:407] Connection header received
0925 04:54:28.537024 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:28.537028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:28.537121 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:28.537231 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:28.538503 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:28.539843 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:28.541186 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:28.541835 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:28.541996 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:28.544260 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:28.544270 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:28.544277 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:28.544315 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:28.548161 (+ 3846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:28.549038 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:28.549043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:28.549046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:28.549201 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:28.549574 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:28.549578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:28.549581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:28.550053 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:28.550200 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:28.550722 (+ 522us) server_negotiation.cc:299] Negotiation successful
0925 04:54:28.551098 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 04:54:28.552330 7734 heartbeater.cc:347] Connected to a master server at 127.6.29.190:33387
I20260925 04:54:28.553012 7734 heartbeater.cc:464] Registering TS with master...
I20260925 04:54:28.553689 7734 heartbeater.cc:511] Master 127.6.29.190:33387 requested a full tablet report, sending...
I20260925 04:54:28.554943 7388 ts_manager.cc:194] Registered new tserver with Master: f1a6d1bf47274f4288760bb24db5d030 (127.6.29.130:44993)
I20260925 04:54:28.555861 7388 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.130@KRBTEST.COM'} at 127.6.29.130:44761
I20260925 04:54:28.556093 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 7599
I20260925 04:54:28.556289 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance
I20260925 04:54:28.556833 6262 external_mini_cluster.cc:1428] Setting key e3d98193b267bc1e0fad6bb62cbeeeb7
WARNING: no policy specified for kudu/127.6.29.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab.
Entry for principal kudu/127.6.29.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab.
I20260925 04:54:28.596237 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.131:0
--local_ip_for_outbound_sockets=127.6.29.131
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:33387
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:44197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:54:28.926781 7738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:54:28.927071 7738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:54:28.927166 7738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:54:28.934715 7738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:54:28.934832 7738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:54:28.935019 7738 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 04:54:28.935148 7738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.131
I20260925 04:54:28.945163 7738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:44197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.29.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.29.190:33387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.131
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:54:28.946471 7738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:54:28.947860 7738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:54:28.961328 7744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:28.961452 7743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:28.961467 7746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:54:28.962584 7738 server_base.cc:1034] running on GCE node
I20260925 04:54:28.963405 7738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:54:28.964542 7738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:54:28.965831 7738 hybrid_clock.cc:648] HybridClock initialized: now 1790312068965765 us; error 41 us; skew 500 ppm
I20260925 04:54:28.966197 7738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:54:28 dist-test-slave-xzjr krb5kdc[7343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312068, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:54:28.969625 7738 init.cc:377] Logged in from keytab as kudu/127.6.29.131@KRBTEST.COM (short username kudu)
I20260925 04:54:28.971442 7738 webserver.cc:460] Webserver started at http://127.6.29.131:44633/ using document root <none> and password file <none>
I20260925 04:54:28.972393 7738 fs_manager.cc:359] Metadata directory not provided
I20260925 04:54:28.972488 7738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:54:28.972820 7738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:54:28.975840 7738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-2/data/instance:
uuid: "169c7739f5ac4977ad1f4c7c303bf653"
format_stamp: "Formatted at 2026-09-25 04:54:28 on dist-test-slave-xzjr"
server_key: "cbe71646bcf45ed67f3cf5a6a2959b5a"
server_key_iv: "5789f5092faf9ed8c6ecba095f6e257e"
server_key_version: "encryptionkey@0"
I20260925 04:54:28.976790 7738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance:
uuid: "169c7739f5ac4977ad1f4c7c303bf653"
format_stamp: "Formatted at 2026-09-25 04:54:28 on dist-test-slave-xzjr"
server_key: "cbe71646bcf45ed67f3cf5a6a2959b5a"
server_key_iv: "5789f5092faf9ed8c6ecba095f6e257e"
server_key_version: "encryptionkey@0"
I20260925 04:54:28.982400 7738 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 04:54:28.986717 7753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:28.988266 7738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:54:28.988474 7738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-2/wal
uuid: "169c7739f5ac4977ad1f4c7c303bf653"
format_stamp: "Formatted at 2026-09-25 04:54:28 on dist-test-slave-xzjr"
server_key: "cbe71646bcf45ed67f3cf5a6a2959b5a"
server_key_iv: "5789f5092faf9ed8c6ecba095f6e257e"
server_key_version: "encryptionkey@0"
I20260925 04:54:28.988696 7738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:54:29.022269 7738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:54:29.027014 7738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:54:29.027309 7738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:54:29.030121 7738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:54:29.030238 7738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:29.030395 7738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:54:29.030473 7738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 04:54:29 dist-test-slave-xzjr krb5kdc[7343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312068, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:54:29.054957 7595 heartbeater.cc:503] Master 127.6.29.190:33387 was elected leader, sending a full tablet report...
I20260925 04:54:29.056576 7483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:29.031017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:54749 (local address 127.6.29.190:33387)
0925 04:54:29.031225 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:29.031230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:29.032861 (+ 1631us) server_negotiation.cc:407] Connection header received
0925 04:54:29.034922 (+ 2061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:29.034927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:29.035024 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:29.035139 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:29.037735 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.039117 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.041819 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.042443 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.042580 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:29.047877 (+ 5297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:29.047885 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:29.047892 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:29.047945 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:29.052283 (+ 4338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:29.053289 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:29.053295 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:29.053298 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:29.053509 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:29.054167 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:29.054175 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:29.054180 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:29.054535 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:29.054717 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:29.055735 (+ 1018us) server_negotiation.cc:299] Negotiation successful
0925 04:54:29.056053 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 04:54:29.057622 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:29.031840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:33387 (local address 127.6.29.131:54749)
0925 04:54:29.032702 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:29.032740 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:29.034643 (+ 1903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:29.035437 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:29.035453 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:29.036412 (+ 959us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:54:29.037404 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:29.037437 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.039346 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.039351 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:29.041589 (+ 2238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:29.041605 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.042995 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.043003 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:29.043317 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:29.044211 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:54:29.044248 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:54:29.047541 (+ 3293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:54:29.052463 (+ 4922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:29.052478 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:29.052499 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:29.053129 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:29.053652 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:29.053660 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:29.053671 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:29.054028 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:29.054697 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:54:29.054709 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:54:29.055213 (+ 504us) client_negotiation.cc:769] Sending connection context
0925 04:54:29.055565 (+ 352us) client_negotiation.cc:240] Negotiation successful
0925 04:54:29.056243 (+ 678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":393,"threads_started":1}
I20260925 04:54:29.075644 7738 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.131:45721
I20260925 04:54:29.075644 7872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.131:45721 every 8 connection(s)
I20260925 04:54:29.076922 7738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
I20260925 04:54:29.079142 7873 heartbeater.cc:347] Connected to a master server at 127.6.29.190:33387
I20260925 04:54:29.079399 7873 heartbeater.cc:464] Registering TS with master...
I20260925 04:54:29.080297 7873 heartbeater.cc:511] Master 127.6.29.190:33387 requested a full tablet report, sending...
I20260925 04:54:29.081806 7850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:29.066546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:33387 (local address 127.6.29.131:57725)
0925 04:54:29.067337 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:29.067361 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:29.067488 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:29.067919 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:29.067924 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:29.068347 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:54:29.068632 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:29.068641 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.070197 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.070203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:29.072029 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:29.072046 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.073549 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.073557 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:29.073738 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:29.074373 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:54:29.074401 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:54:29.075702 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:54:29.079234 (+ 3532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:29.079239 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:29.079242 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:29.079815 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:29.080164 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:29.080169 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:29.080172 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:29.080328 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:29.080858 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:54:29.080863 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:54:29.081049 (+ 186us) client_negotiation.cc:769] Sending connection context
0925 04:54:29.081208 (+ 159us) client_negotiation.cc:240] Negotiation successful
0925 04:54:29.081514 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":502,"threads_started":1}
I20260925 04:54:29.081871 7387 ts_manager.cc:194] Registered new tserver with Master: 169c7739f5ac4977ad1f4c7c303bf653 (127.6.29.131:45721)
I20260925 04:54:29.081928 7483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:29.066418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:57725 (local address 127.6.29.190:33387)
0925 04:54:29.066638 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:29.066647 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:29.067466 (+ 819us) server_negotiation.cc:407] Connection header received
0925 04:54:29.067611 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:29.067619 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:29.067722 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:29.067862 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:29.068773 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.070048 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.072194 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.073009 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.073151 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:29.075988 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:29.075994 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:29.075999 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:29.076040 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:29.079095 (+ 3055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:29.079920 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:29.079924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:29.079927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:29.080047 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:29.080453 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:29.080458 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:29.080461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:29.080732 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:29.080839 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:29.081340 (+ 501us) server_negotiation.cc:299] Negotiation successful
0925 04:54:29.081560 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 04:54:29.082775 7387 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.131@KRBTEST.COM'} at 127.6.29.131:54749
I20260925 04:54:29.084213 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 7738
I20260925 04:54:29.084393 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance
I20260925 04:54:29.084990 6262 external_mini_cluster.cc:1428] Setting key e1cd3c6c96de74fc5516df8c88bfb170
I20260925 04:54:29.087831 6262 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 04:54:29 dist-test-slave-xzjr krb5kdc[7343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312069, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 04:54:29 dist-test-slave-xzjr krb5kdc[7343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312069, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:54:29.133464 7483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:29.119412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53166 (local address 127.6.29.190:33387)
0925 04:54:29.119640 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:29.119647 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:29.120046 (+ 399us) server_negotiation.cc:407] Connection header received
0925 04:54:29.120114 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:29.120118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:29.120184 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:29.120283 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:29.121593 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.122390 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.123706 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.124241 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.124350 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:29.128260 (+ 3910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:29.128265 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:29.128268 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:29.128302 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:29.130916 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:29.131553 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:29.131565 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:29.131567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:29.131683 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:29.132081 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:29.132088 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:29.132091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:29.132362 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:29.132601 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:29.132919 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 04:54:29.133134 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 04:54:29.137488 7388 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53166:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:54:29.144447 7388 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:54:29.178296 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:29.159224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:41377 (local address 127.0.0.1:58382)
0925 04:54:29.170926 (+ 11702us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:29.170948 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:29.171072 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:29.171824 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:29.171828 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:29.171854 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:54:29.172194 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:29.172202 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.173941 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.173946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:29.175821 (+ 1875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:29.175835 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.177308 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.177314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:29.177509 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:29.177556 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 04:54:29.177690 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 04:54:29.177877 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11566,"mutex_wait_us":101,"thread_start_us":9908,"threads_started":1}
I20260925 04:54:29.178287 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:29.159805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58382 (local address 127.6.29.129:41377)
0925 04:54:29.170497 (+ 10692us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:29.170508 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:29.171040 (+ 532us) server_negotiation.cc:407] Connection header received
0925 04:54:29.171245 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:29.171252 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:29.171475 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:29.171658 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:54:29.172360 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.173769 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.175971 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.177120 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.177257 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:29.177464 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:29.177767 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 04:54:29.177995 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10533,"thread_start_us":10389,"threads_started":1}
I20260925 04:54:29.179901 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:29.156580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.131:45721 (local address 127.0.0.1:51578)
0925 04:54:29.168480 (+ 11900us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:29.168531 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:29.169572 (+ 1041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:29.170733 (+ 1161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:29.170744 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:29.170782 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:54:29.171324 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:29.171334 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.173526 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.173531 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:29.176475 (+ 2944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:29.176489 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.178385 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.178390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:29.178553 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:29.178601 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 04:54:29.178700 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 04:54:29.178851 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11550,"mutex_wait_us":297,"thread_start_us":14062,"threads_started":1}
I20260925 04:54:29.179951 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:29.156827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:44993 (local address 127.0.0.1:34990)
0925 04:54:29.169426 (+ 12599us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:29.169445 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:29.169566 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:29.170929 (+ 1363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:29.170937 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:29.170981 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:54:29.171553 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:29.171572 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.173610 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.173617 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:29.176479 (+ 2862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:29.176489 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.178594 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.178600 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:29.178992 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:29.179024 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 04:54:29.179159 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 04:54:29.179276 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12484,"mutex_wait_us":223,"spinlock_wait_cycles":5888,"thread_start_us":11874,"threads_started":1}
I20260925 04:54:29.180482 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:29.157018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34990 (local address 127.6.29.130:44993)
0925 04:54:29.170205 (+ 13187us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:29.170220 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:29.170247 (+ 27us) server_negotiation.cc:407] Connection header received
0925 04:54:29.170333 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:29.170339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:29.170617 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:29.170820 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:54:29.171794 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.173448 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.176641 (+ 3193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.177927 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.178089 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:29.178282 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:29.179743 (+ 1461us) server_negotiation.cc:299] Negotiation successful
0925 04:54:29.179941 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12994,"thread_start_us":12861,"threads_started":1}
I20260925 04:54:29.182036 7525 tablet_service.cc:1467] Processing CreateTablet for tablet d4120d0ad42b450f9706dc0090899337 (DEFAULT_TABLE table=test-table [id=b3727645652a4d5d8965138a2fc50af2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:54:29.183257 7525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4120d0ad42b450f9706dc0090899337. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:54:29.183528 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:29.157367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51578 (local address 127.6.29.131:45721)
0925 04:54:29.158353 (+ 986us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:29.158368 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:29.168661 (+ 10293us) server_negotiation.cc:407] Connection header received
0925 04:54:29.169877 (+ 1216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:29.169886 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:29.170154 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:29.170372 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:54:29.171575 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.173352 (+ 1777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.176657 (+ 3305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.177838 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.177950 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:29.178235 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:29.179368 (+ 1133us) server_negotiation.cc:299] Negotiation successful
0925 04:54:29.179531 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":769,"thread_start_us":483,"threads_started":1}
I20260925 04:54:29.185796 7664 tablet_service.cc:1467] Processing CreateTablet for tablet d4120d0ad42b450f9706dc0090899337 (DEFAULT_TABLE table=test-table [id=b3727645652a4d5d8965138a2fc50af2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:54:29.186303 7803 tablet_service.cc:1467] Processing CreateTablet for tablet d4120d0ad42b450f9706dc0090899337 (DEFAULT_TABLE table=test-table [id=b3727645652a4d5d8965138a2fc50af2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:54:29.187189 7664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4120d0ad42b450f9706dc0090899337. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:54:29.187556 7803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4120d0ad42b450f9706dc0090899337. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:54:29.198088 7892 tablet_bootstrap.cc:492] T d4120d0ad42b450f9706dc0090899337 P cc3aa37d80bd4be88566dc19585eef45: Bootstrap starting.
I20260925 04:54:29.200862 7894 tablet_bootstrap.cc:492] T d4120d0ad42b450f9706dc0090899337 P f1a6d1bf47274f4288760bb24db5d030: Bootstrap starting.
I20260925 04:54:29.203440 7894 tablet_bootstrap.cc:654] T d4120d0ad42b450f9706dc0090899337 P f1a6d1bf47274f4288760bb24db5d030: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:29.203588 7892 tablet_bootstrap.cc:654] T d4120d0ad42b450f9706dc0090899337 P cc3aa37d80bd4be88566dc19585eef45: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:29.204345 7893 tablet_bootstrap.cc:492] T d4120d0ad42b450f9706dc0090899337 P 169c7739f5ac4977ad1f4c7c303bf653: Bootstrap starting.
I20260925 04:54:29.204770 7894 log.cc:824] T d4120d0ad42b450f9706dc0090899337 P f1a6d1bf47274f4288760bb24db5d030: Log is configured to *not* fsync() on all Append() calls
I20260925 04:54:29.205958 7892 log.cc:824] T d4120d0ad42b450f9706dc0090899337 P cc3aa37d80bd4be88566dc19585eef45: Log is configured to *not* fsync() on all Append() calls
I20260925 04:54:29.207998 7893 tablet_bootstrap.cc:654] T d4120d0ad42b450f9706dc0090899337 P 169c7739f5ac4977ad1f4c7c303bf653: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:29.208858 7894 tablet_bootstrap.cc:492] T d4120d0ad42b450f9706dc0090899337 P f1a6d1bf47274f4288760bb24db5d030: No bootstrap required, opened a new log
I20260925 04:54:29.209088 7892 tablet_bootstrap.cc:492] T d4120d0ad42b450f9706dc0090899337 P cc3aa37d80bd4be88566dc19585eef45: No bootstrap required, opened a new log
I20260925 04:54:29.209118 7894 ts_tablet_manager.cc:1397] T d4120d0ad42b450f9706dc0090899337 P f1a6d1bf47274f4288760bb24db5d030: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.001s
I20260925 04:54:29.209209 7893 log.cc:824] T d4120d0ad42b450f9706dc0090899337 P 169c7739f5ac4977ad1f4c7c303bf653: Log is configured to *not* fsync() on all Append() calls
I20260925 04:54:29.209353 7892 ts_tablet_manager.cc:1397] T d4120d0ad42b450f9706dc0090899337 P cc3aa37d80bd4be88566dc19585eef45: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260925 04:54:29.214733 7893 tablet_bootstrap.cc:492] T d4120d0ad42b450f9706dc0090899337 P 169c7739f5ac4977ad1f4c7c303bf653: No bootstrap required, opened a new log
I20260925 04:54:29.215049 7893 ts_tablet_manager.cc:1397] T d4120d0ad42b450f9706dc0090899337 P 169c7739f5ac4977ad1f4c7c303bf653: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.008s
I20260925 04:54:29.217308 7894 raft_consensus.cc:348] T d4120d0ad42b450f9706dc0090899337 P f1a6d1bf47274f4288760bb24db5d030 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169c7739f5ac4977ad1f4c7c303bf653" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 45721 } } peers { permanent_uuid: "f1a6d1bf47274f4288760bb24db5d030" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44993 } } peers { permanent_uuid: "cc3aa37d80bd4be88566dc19585eef45" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41377 } }
I20260925 04:54:29.217728 7894 raft_consensus.cc:374] T d4120d0ad42b450f9706dc0090899337 P f1a6d1bf47274f4288760bb24db5d030 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:29.217835 7894 raft_consensus.cc:729] T d4120d0ad42b450f9706dc0090899337 P f1a6d1bf47274f4288760bb24db5d030 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a6d1bf47274f4288760bb24db5d030, State: Initialized, Role: FOLLOWER
I20260925 04:54:29.218925 7894 consensus_queue.cc:260] T d4120d0ad42b450f9706dc0090899337 P f1a6d1bf47274f4288760bb24db5d030 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169c7739f5ac4977ad1f4c7c303bf653" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 45721 } } peers { permanent_uuid: "f1a6d1bf47274f4288760bb24db5d030" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44993 } } peers { permanent_uuid: "cc3aa37d80bd4be88566dc19585eef45" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41377 } }
I20260925 04:54:29.220516 7734 heartbeater.cc:503] Master 127.6.29.190:33387 was elected leader, sending a full tablet report...
I20260925 04:54:29.220638 7892 raft_consensus.cc:348] T d4120d0ad42b450f9706dc0090899337 P cc3aa37d80bd4be88566dc19585eef45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169c7739f5ac4977ad1f4c7c303bf653" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 45721 } } peers { permanent_uuid: "f1a6d1bf47274f4288760bb24db5d030" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44993 } } peers { permanent_uuid: "cc3aa37d80bd4be88566dc19585eef45" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41377 } }
I20260925 04:54:29.221064 7892 raft_consensus.cc:374] T d4120d0ad42b450f9706dc0090899337 P cc3aa37d80bd4be88566dc19585eef45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:29.221148 7892 raft_consensus.cc:729] T d4120d0ad42b450f9706dc0090899337 P cc3aa37d80bd4be88566dc19585eef45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc3aa37d80bd4be88566dc19585eef45, State: Initialized, Role: FOLLOWER
I20260925 04:54:29.221251 7894 ts_tablet_manager.cc:1428] T d4120d0ad42b450f9706dc0090899337 P f1a6d1bf47274f4288760bb24db5d030: Time spent starting tablet: real 0.012s user 0.006s sys 0.005s
I20260925 04:54:29.221779 7892 consensus_queue.cc:260] T d4120d0ad42b450f9706dc0090899337 P cc3aa37d80bd4be88566dc19585eef45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169c7739f5ac4977ad1f4c7c303bf653" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 45721 } } peers { permanent_uuid: "f1a6d1bf47274f4288760bb24db5d030" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44993 } } peers { permanent_uuid: "cc3aa37d80bd4be88566dc19585eef45" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41377 } }
I20260925 04:54:29.225589 7892 ts_tablet_manager.cc:1428] T d4120d0ad42b450f9706dc0090899337 P cc3aa37d80bd4be88566dc19585eef45: Time spent starting tablet: real 0.016s user 0.009s sys 0.007s
I20260925 04:54:29.226711 7893 raft_consensus.cc:348] T d4120d0ad42b450f9706dc0090899337 P 169c7739f5ac4977ad1f4c7c303bf653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169c7739f5ac4977ad1f4c7c303bf653" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 45721 } } peers { permanent_uuid: "f1a6d1bf47274f4288760bb24db5d030" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44993 } } peers { permanent_uuid: "cc3aa37d80bd4be88566dc19585eef45" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41377 } }
I20260925 04:54:29.227238 7893 raft_consensus.cc:374] T d4120d0ad42b450f9706dc0090899337 P 169c7739f5ac4977ad1f4c7c303bf653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:29.227362 7893 raft_consensus.cc:729] T d4120d0ad42b450f9706dc0090899337 P 169c7739f5ac4977ad1f4c7c303bf653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 169c7739f5ac4977ad1f4c7c303bf653, State: Initialized, Role: FOLLOWER
I20260925 04:54:29.228014 7893 consensus_queue.cc:260] T d4120d0ad42b450f9706dc0090899337 P 169c7739f5ac4977ad1f4c7c303bf653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169c7739f5ac4977ad1f4c7c303bf653" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 45721 } } peers { permanent_uuid: "f1a6d1bf47274f4288760bb24db5d030" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44993 } } peers { permanent_uuid: "cc3aa37d80bd4be88566dc19585eef45" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41377 } }
I20260925 04:54:29.229187 7873 heartbeater.cc:503] Master 127.6.29.190:33387 was elected leader, sending a full tablet report...
I20260925 04:54:29.229871 7893 ts_tablet_manager.cc:1428] T d4120d0ad42b450f9706dc0090899337 P 169c7739f5ac4977ad1f4c7c303bf653: Time spent starting tablet: real 0.015s user 0.014s sys 0.002s
W20260925 04:54:29.296649 7596 tablet.cc:2369] T d4120d0ad42b450f9706dc0090899337 P cc3aa37d80bd4be88566dc19585eef45: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:54:29.303491 7735 tablet.cc:2369] T d4120d0ad42b450f9706dc0090899337 P f1a6d1bf47274f4288760bb24db5d030: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:54:29.328761 7874 tablet.cc:2369] T d4120d0ad42b450f9706dc0090899337 P 169c7739f5ac4977ad1f4c7c303bf653: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:54:29.367560 7900 raft_consensus.cc:482] T d4120d0ad42b450f9706dc0090899337 P 169c7739f5ac4977ad1f4c7c303bf653 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:54:29.367937 7900 raft_consensus.cc:504] T d4120d0ad42b450f9706dc0090899337 P 169c7739f5ac4977ad1f4c7c303bf653 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169c7739f5ac4977ad1f4c7c303bf653" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 45721 } } peers { permanent_uuid: "f1a6d1bf47274f4288760bb24db5d030" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44993 } } peers { permanent_uuid: "cc3aa37d80bd4be88566dc19585eef45" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41377 } }
I20260925 04:54:29.369557 7900 leader_election.cc:288] T d4120d0ad42b450f9706dc0090899337 P 169c7739f5ac4977ad1f4c7c303bf653 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1a6d1bf47274f4288760bb24db5d030 (127.6.29.130:44993), cc3aa37d80bd4be88566dc19585eef45 (127.6.29.129:41377)
I20260925 04:54:29.375972 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:29.369943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:41377 (local address 127.6.29.131:32947)
0925 04:54:29.370281 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:29.370308 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:29.370496 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:29.370935 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:29.370940 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:29.370974 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:54:29.371175 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:29.371187 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.372396 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.372402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:29.374060 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:29.374071 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.375211 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.375218 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:29.375428 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:29.375460 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 04:54:29.375545 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 04:54:29.375634 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":66}
I20260925 04:54:29.375995 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:29.370133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:32947 (local address 127.6.29.129:41377)
0925 04:54:29.370328 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:29.370336 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:29.370423 (+ 87us) server_negotiation.cc:407] Connection header received
0925 04:54:29.370662 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:29.370667 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:29.370760 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:29.370866 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:54:29.371332 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.372260 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.374221 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.375048 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.375170 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:29.375221 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:29.375599 (+ 378us) server_negotiation.cc:299] Negotiation successful
0925 04:54:29.375742 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 04:54:29.376139 7888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:29.370042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:57809 (local address 127.6.29.130:44993)
0925 04:54:29.370241 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:29.370248 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:29.370900 (+ 652us) server_negotiation.cc:407] Connection header received
0925 04:54:29.371034 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:29.371039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:29.371115 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:29.371235 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:54:29.371827 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.372662 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.374419 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.375201 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.375388 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:29.375454 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:29.375740 (+ 286us) server_negotiation.cc:299] Negotiation successful
0925 04:54:29.375887 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 04:54:29.376660 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:29.369972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:44993 (local address 127.6.29.131:57809)
0925 04:54:29.370741 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:29.370759 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:29.370894 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:29.371239 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:29.371264 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:29.371290 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:54:29.371646 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:29.371657 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.372816 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.372822 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:29.374270 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:29.374283 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.375427 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.375431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:29.375580 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:29.375611 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 04:54:29.375698 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 04:54:29.375845 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"mutex_wait_us":1,"thread_start_us":451,"threads_started":1}
I20260925 04:54:29.377187 7550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4120d0ad42b450f9706dc0090899337" candidate_uuid: "169c7739f5ac4977ad1f4c7c303bf653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc3aa37d80bd4be88566dc19585eef45" is_pre_election: true
I20260925 04:54:29.377698 7550 raft_consensus.cc:2454] T d4120d0ad42b450f9706dc0090899337 P cc3aa37d80bd4be88566dc19585eef45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 169c7739f5ac4977ad1f4c7c303bf653 in term 0.
I20260925 04:54:29.377677 7684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4120d0ad42b450f9706dc0090899337" candidate_uuid: "169c7739f5ac4977ad1f4c7c303bf653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a6d1bf47274f4288760bb24db5d030" is_pre_election: true
I20260925 04:54:29.378320 7684 raft_consensus.cc:2454] T d4120d0ad42b450f9706dc0090899337 P f1a6d1bf47274f4288760bb24db5d030 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 169c7739f5ac4977ad1f4c7c303bf653 in term 0.
I20260925 04:54:29.378365 7756 leader_election.cc:302] T d4120d0ad42b450f9706dc0090899337 P 169c7739f5ac4977ad1f4c7c303bf653 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 169c7739f5ac4977ad1f4c7c303bf653, cc3aa37d80bd4be88566dc19585eef45; no voters:
I20260925 04:54:29.378731 7900 raft_consensus.cc:2781] T d4120d0ad42b450f9706dc0090899337 P 169c7739f5ac4977ad1f4c7c303bf653 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:54:29.378885 7900 raft_consensus.cc:482] T d4120d0ad42b450f9706dc0090899337 P 169c7739f5ac4977ad1f4c7c303bf653 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:54:29.379015 7900 raft_consensus.cc:3037] T d4120d0ad42b450f9706dc0090899337 P 169c7739f5ac4977ad1f4c7c303bf653 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:29.381333 7900 raft_consensus.cc:504] T d4120d0ad42b450f9706dc0090899337 P 169c7739f5ac4977ad1f4c7c303bf653 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169c7739f5ac4977ad1f4c7c303bf653" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 45721 } } peers { permanent_uuid: "f1a6d1bf47274f4288760bb24db5d030" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44993 } } peers { permanent_uuid: "cc3aa37d80bd4be88566dc19585eef45" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41377 } }
I20260925 04:54:29.381907 7900 leader_election.cc:288] T d4120d0ad42b450f9706dc0090899337 P 169c7739f5ac4977ad1f4c7c303bf653 [CANDIDATE]: Term 1 election: Requested vote from peers f1a6d1bf47274f4288760bb24db5d030 (127.6.29.130:44993), cc3aa37d80bd4be88566dc19585eef45 (127.6.29.129:41377)
I20260925 04:54:29.382368 7684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4120d0ad42b450f9706dc0090899337" candidate_uuid: "169c7739f5ac4977ad1f4c7c303bf653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a6d1bf47274f4288760bb24db5d030"
I20260925 04:54:29.382366 7550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4120d0ad42b450f9706dc0090899337" candidate_uuid: "169c7739f5ac4977ad1f4c7c303bf653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc3aa37d80bd4be88566dc19585eef45"
I20260925 04:54:29.382568 7684 raft_consensus.cc:3037] T d4120d0ad42b450f9706dc0090899337 P f1a6d1bf47274f4288760bb24db5d030 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:29.382568 7550 raft_consensus.cc:3037] T d4120d0ad42b450f9706dc0090899337 P cc3aa37d80bd4be88566dc19585eef45 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:29.385731 7684 raft_consensus.cc:2454] T d4120d0ad42b450f9706dc0090899337 P f1a6d1bf47274f4288760bb24db5d030 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 169c7739f5ac4977ad1f4c7c303bf653 in term 1.
I20260925 04:54:29.385731 7550 raft_consensus.cc:2454] T d4120d0ad42b450f9706dc0090899337 P cc3aa37d80bd4be88566dc19585eef45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 169c7739f5ac4977ad1f4c7c303bf653 in term 1.
I20260925 04:54:29.386139 7757 leader_election.cc:302] T d4120d0ad42b450f9706dc0090899337 P 169c7739f5ac4977ad1f4c7c303bf653 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 169c7739f5ac4977ad1f4c7c303bf653, f1a6d1bf47274f4288760bb24db5d030; no voters:
I20260925 04:54:29.386505 7900 raft_consensus.cc:2781] T d4120d0ad42b450f9706dc0090899337 P 169c7739f5ac4977ad1f4c7c303bf653 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:54:29.386956 7900 raft_consensus.cc:686] T d4120d0ad42b450f9706dc0090899337 P 169c7739f5ac4977ad1f4c7c303bf653 [term 1 LEADER]: Becoming Leader. State: Replica: 169c7739f5ac4977ad1f4c7c303bf653, State: Running, Role: LEADER
I20260925 04:54:29.387449 7900 consensus_queue.cc:237] T d4120d0ad42b450f9706dc0090899337 P 169c7739f5ac4977ad1f4c7c303bf653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169c7739f5ac4977ad1f4c7c303bf653" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 45721 } } peers { permanent_uuid: "f1a6d1bf47274f4288760bb24db5d030" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44993 } } peers { permanent_uuid: "cc3aa37d80bd4be88566dc19585eef45" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41377 } }
I20260925 04:54:29.392499 7387 catalog_manager.cc:5495] T d4120d0ad42b450f9706dc0090899337 P 169c7739f5ac4977ad1f4c7c303bf653 reported cstate change: term changed from 0 to 1, leader changed from <none> to 169c7739f5ac4977ad1f4c7c303bf653 (127.6.29.131). New cstate: current_term: 1 leader_uuid: "169c7739f5ac4977ad1f4c7c303bf653" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "169c7739f5ac4977ad1f4c7c303bf653" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 45721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a6d1bf47274f4288760bb24db5d030" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc3aa37d80bd4be88566dc19585eef45" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41377 } health_report { overall_health: UNKNOWN } } }
I20260925 04:54:29.467897 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:29.462478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57360 (local address 127.6.29.131:45721)
0925 04:54:29.462695 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:29.462703 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:29.462829 (+ 126us) server_negotiation.cc:407] Connection header received
0925 04:54:29.463212 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:29.463216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:29.463292 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:29.463444 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:54:29.463836 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.464639 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.465871 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.466250 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.466345 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:29.466773 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:54:29.467258 (+ 485us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:54:29.467375 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:29.467488 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 04:54:29.467549 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 04:54:29.475742 7550 raft_consensus.cc:1261] T d4120d0ad42b450f9706dc0090899337 P cc3aa37d80bd4be88566dc19585eef45 [term 1 FOLLOWER]: Refusing update from remote peer 169c7739f5ac4977ad1f4c7c303bf653: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:54:29.475742 7684 raft_consensus.cc:1261] T d4120d0ad42b450f9706dc0090899337 P f1a6d1bf47274f4288760bb24db5d030 [term 1 FOLLOWER]: Refusing update from remote peer 169c7739f5ac4977ad1f4c7c303bf653: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:54:29.476802 7902 consensus_queue.cc:1035] T d4120d0ad42b450f9706dc0090899337 P 169c7739f5ac4977ad1f4c7c303bf653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a6d1bf47274f4288760bb24db5d030" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:54:29.477205 7900 consensus_queue.cc:1035] T d4120d0ad42b450f9706dc0090899337 P 169c7739f5ac4977ad1f4c7c303bf653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3aa37d80bd4be88566dc19585eef45" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:54:29.492148 7907 mvcc.cc:205] Tried to move back new op lower bound from 7333118236562579456 to 7333118236214726656. Current Snapshot: MvccSnapshot[applied={T|T < 7333118236562579456}]
I20260925 04:54:29.495860 7905 mvcc.cc:205] Tried to move back new op lower bound from 7333118236562579456 to 7333118236214726656. Current Snapshot: MvccSnapshot[applied={T|T < 7333118236562579456 or (T in {7333118236562579456})}]
I20260925 04:54:29.510664 7387 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:54:29.510931 7387 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:54:29.514472 7387 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b60b88bcac904dd584ad8b377a235dc1: Sending DeleteTablet for 3 replicas of tablet d4120d0ad42b450f9706dc0090899337
I20260925 04:54:29.515595 7803 tablet_service.cc:1514] Processing DeleteTablet for tablet d4120d0ad42b450f9706dc0090899337 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:54:29 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:51578
I20260925 04:54:29.515664 7664 tablet_service.cc:1514] Processing DeleteTablet for tablet d4120d0ad42b450f9706dc0090899337 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:54:29 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:34990
I20260925 04:54:29.516254 7525 tablet_service.cc:1514] Processing DeleteTablet for tablet d4120d0ad42b450f9706dc0090899337 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:54:29 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:58382
I20260925 04:54:29.516636 7915 tablet_replica.cc:330] T d4120d0ad42b450f9706dc0090899337 P 169c7739f5ac4977ad1f4c7c303bf653: stopping tablet replica
I20260925 04:54:29.517055 7915 raft_consensus.cc:2229] T d4120d0ad42b450f9706dc0090899337 P 169c7739f5ac4977ad1f4c7c303bf653 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:54:29.517696 7915 raft_consensus.cc:2258] T d4120d0ad42b450f9706dc0090899337 P 169c7739f5ac4977ad1f4c7c303bf653 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:54:29.518527 7387 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53166:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:54:29.519169 7387 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:54:29.519939 7915 ts_tablet_manager.cc:1905] T d4120d0ad42b450f9706dc0090899337 P 169c7739f5ac4977ad1f4c7c303bf653: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:54:29.526912 7915 ts_tablet_manager.cc:1918] T d4120d0ad42b450f9706dc0090899337 P 169c7739f5ac4977ad1f4c7c303bf653: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:54:29.527077 7915 log.cc:1196] T d4120d0ad42b450f9706dc0090899337 P 169c7739f5ac4977ad1f4c7c303bf653: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-2/wal/wals/d4120d0ad42b450f9706dc0090899337
I20260925 04:54:29.527611 7915 ts_tablet_manager.cc:1939] T d4120d0ad42b450f9706dc0090899337 P 169c7739f5ac4977ad1f4c7c303bf653: Deleting consensus metadata
I20260925 04:54:29.528748 7663 tablet_service.cc:1467] Processing CreateTablet for tablet b982823b08cf42e5998a9888c4728626 (DEFAULT_TABLE table=test-table [id=e711c99d71f249f983377afd45ec7a06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:54:29.529336 7663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b982823b08cf42e5998a9888c4728626. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:54:29.529724 7524 tablet_service.cc:1467] Processing CreateTablet for tablet b982823b08cf42e5998a9888c4728626 (DEFAULT_TABLE table=test-table [id=e711c99d71f249f983377afd45ec7a06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:54:29.530304 7524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b982823b08cf42e5998a9888c4728626. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:54:29.531417 7916 tablet_replica.cc:330] T d4120d0ad42b450f9706dc0090899337 P f1a6d1bf47274f4288760bb24db5d030: stopping tablet replica
I20260925 04:54:29.531975 7916 raft_consensus.cc:2229] T d4120d0ad42b450f9706dc0090899337 P f1a6d1bf47274f4288760bb24db5d030 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:54:29.531961 7373 catalog_manager.cc:4841] TS 169c7739f5ac4977ad1f4c7c303bf653 (127.6.29.131:45721): tablet d4120d0ad42b450f9706dc0090899337 (table test-table [id=b3727645652a4d5d8965138a2fc50af2]) successfully deleted
I20260925 04:54:29.532528 7803 tablet_service.cc:1467] Processing CreateTablet for tablet b982823b08cf42e5998a9888c4728626 (DEFAULT_TABLE table=test-table [id=e711c99d71f249f983377afd45ec7a06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:54:29.532974 7803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b982823b08cf42e5998a9888c4728626. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:54:29.533556 7916 raft_consensus.cc:2258] T d4120d0ad42b450f9706dc0090899337 P f1a6d1bf47274f4288760bb24db5d030 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:54:29.534368 7917 tablet_replica.cc:330] T d4120d0ad42b450f9706dc0090899337 P cc3aa37d80bd4be88566dc19585eef45: stopping tablet replica
I20260925 04:54:29.534876 7917 raft_consensus.cc:2229] T d4120d0ad42b450f9706dc0090899337 P cc3aa37d80bd4be88566dc19585eef45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:54:29.535374 7917 raft_consensus.cc:2258] T d4120d0ad42b450f9706dc0090899337 P cc3aa37d80bd4be88566dc19585eef45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:54:29.537350 7892 tablet_bootstrap.cc:492] T b982823b08cf42e5998a9888c4728626 P cc3aa37d80bd4be88566dc19585eef45: Bootstrap starting.
I20260925 04:54:29.538748 7917 ts_tablet_manager.cc:1905] T d4120d0ad42b450f9706dc0090899337 P cc3aa37d80bd4be88566dc19585eef45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:54:29.539077 7892 tablet_bootstrap.cc:654] T b982823b08cf42e5998a9888c4728626 P cc3aa37d80bd4be88566dc19585eef45: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:29.540597 7894 tablet_bootstrap.cc:492] T b982823b08cf42e5998a9888c4728626 P f1a6d1bf47274f4288760bb24db5d030: Bootstrap starting.
I20260925 04:54:29.541802 7893 tablet_bootstrap.cc:492] T b982823b08cf42e5998a9888c4728626 P 169c7739f5ac4977ad1f4c7c303bf653: Bootstrap starting.
I20260925 04:54:29.542253 7894 tablet_bootstrap.cc:654] T b982823b08cf42e5998a9888c4728626 P f1a6d1bf47274f4288760bb24db5d030: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:29.543902 7893 tablet_bootstrap.cc:654] T b982823b08cf42e5998a9888c4728626 P 169c7739f5ac4977ad1f4c7c303bf653: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:29.544518 7916 ts_tablet_manager.cc:1905] T d4120d0ad42b450f9706dc0090899337 P f1a6d1bf47274f4288760bb24db5d030: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:54:29.553057 7894 tablet_bootstrap.cc:492] T b982823b08cf42e5998a9888c4728626 P f1a6d1bf47274f4288760bb24db5d030: No bootstrap required, opened a new log
I20260925 04:54:29.553217 7894 ts_tablet_manager.cc:1397] T b982823b08cf42e5998a9888c4728626 P f1a6d1bf47274f4288760bb24db5d030: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.006s
I20260925 04:54:29.553755 7916 ts_tablet_manager.cc:1918] T d4120d0ad42b450f9706dc0090899337 P f1a6d1bf47274f4288760bb24db5d030: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:54:29.554029 7916 log.cc:1196] T d4120d0ad42b450f9706dc0090899337 P f1a6d1bf47274f4288760bb24db5d030: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-1/wal/wals/d4120d0ad42b450f9706dc0090899337
I20260925 04:54:29.554129 7894 raft_consensus.cc:348] T b982823b08cf42e5998a9888c4728626 P f1a6d1bf47274f4288760bb24db5d030 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3aa37d80bd4be88566dc19585eef45" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41377 } } peers { permanent_uuid: "f1a6d1bf47274f4288760bb24db5d030" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44993 } } peers { permanent_uuid: "169c7739f5ac4977ad1f4c7c303bf653" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 45721 } }
I20260925 04:54:29.554406 7894 raft_consensus.cc:374] T b982823b08cf42e5998a9888c4728626 P f1a6d1bf47274f4288760bb24db5d030 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:29.554512 7894 raft_consensus.cc:729] T b982823b08cf42e5998a9888c4728626 P f1a6d1bf47274f4288760bb24db5d030 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a6d1bf47274f4288760bb24db5d030, State: Initialized, Role: FOLLOWER
I20260925 04:54:29.554929 7916 ts_tablet_manager.cc:1939] T d4120d0ad42b450f9706dc0090899337 P f1a6d1bf47274f4288760bb24db5d030: Deleting consensus metadata
I20260925 04:54:29.554872 7894 consensus_queue.cc:260] T b982823b08cf42e5998a9888c4728626 P f1a6d1bf47274f4288760bb24db5d030 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3aa37d80bd4be88566dc19585eef45" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41377 } } peers { permanent_uuid: "f1a6d1bf47274f4288760bb24db5d030" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44993 } } peers { permanent_uuid: "169c7739f5ac4977ad1f4c7c303bf653" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 45721 } }
I20260925 04:54:29.557391 7894 ts_tablet_manager.cc:1428] T b982823b08cf42e5998a9888c4728626 P f1a6d1bf47274f4288760bb24db5d030: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260925 04:54:29.557569 7375 catalog_manager.cc:4841] TS f1a6d1bf47274f4288760bb24db5d030 (127.6.29.130:44993): tablet d4120d0ad42b450f9706dc0090899337 (table test-table [id=b3727645652a4d5d8965138a2fc50af2]) successfully deleted
I20260925 04:54:29.561640 7893 tablet_bootstrap.cc:492] T b982823b08cf42e5998a9888c4728626 P 169c7739f5ac4977ad1f4c7c303bf653: No bootstrap required, opened a new log
I20260925 04:54:29.561877 7893 ts_tablet_manager.cc:1397] T b982823b08cf42e5998a9888c4728626 P 169c7739f5ac4977ad1f4c7c303bf653: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.005s
I20260925 04:54:29.562203 7917 ts_tablet_manager.cc:1918] T d4120d0ad42b450f9706dc0090899337 P cc3aa37d80bd4be88566dc19585eef45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:54:29.562443 7917 log.cc:1196] T d4120d0ad42b450f9706dc0090899337 P cc3aa37d80bd4be88566dc19585eef45: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-0/wal/wals/d4120d0ad42b450f9706dc0090899337
I20260925 04:54:29.562711 7893 raft_consensus.cc:348] T b982823b08cf42e5998a9888c4728626 P 169c7739f5ac4977ad1f4c7c303bf653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3aa37d80bd4be88566dc19585eef45" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41377 } } peers { permanent_uuid: "f1a6d1bf47274f4288760bb24db5d030" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44993 } } peers { permanent_uuid: "169c7739f5ac4977ad1f4c7c303bf653" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 45721 } }
I20260925 04:54:29.562933 7893 raft_consensus.cc:374] T b982823b08cf42e5998a9888c4728626 P 169c7739f5ac4977ad1f4c7c303bf653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:29.563021 7893 raft_consensus.cc:729] T b982823b08cf42e5998a9888c4728626 P 169c7739f5ac4977ad1f4c7c303bf653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 169c7739f5ac4977ad1f4c7c303bf653, State: Initialized, Role: FOLLOWER
I20260925 04:54:29.563225 7917 ts_tablet_manager.cc:1939] T d4120d0ad42b450f9706dc0090899337 P cc3aa37d80bd4be88566dc19585eef45: Deleting consensus metadata
I20260925 04:54:29.563318 7893 consensus_queue.cc:260] T b982823b08cf42e5998a9888c4728626 P 169c7739f5ac4977ad1f4c7c303bf653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3aa37d80bd4be88566dc19585eef45" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41377 } } peers { permanent_uuid: "f1a6d1bf47274f4288760bb24db5d030" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44993 } } peers { permanent_uuid: "169c7739f5ac4977ad1f4c7c303bf653" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 45721 } }
I20260925 04:54:29.563876 7893 ts_tablet_manager.cc:1428] T b982823b08cf42e5998a9888c4728626 P 169c7739f5ac4977ad1f4c7c303bf653: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260925 04:54:29.565514 7892 tablet_bootstrap.cc:492] T b982823b08cf42e5998a9888c4728626 P cc3aa37d80bd4be88566dc19585eef45: No bootstrap required, opened a new log
I20260925 04:54:29.565554 7374 catalog_manager.cc:4841] TS cc3aa37d80bd4be88566dc19585eef45 (127.6.29.129:41377): tablet d4120d0ad42b450f9706dc0090899337 (table test-table [id=b3727645652a4d5d8965138a2fc50af2]) successfully deleted
I20260925 04:54:29.565656 7892 ts_tablet_manager.cc:1397] T b982823b08cf42e5998a9888c4728626 P cc3aa37d80bd4be88566dc19585eef45: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.004s
I20260925 04:54:29.566900 7892 raft_consensus.cc:348] T b982823b08cf42e5998a9888c4728626 P cc3aa37d80bd4be88566dc19585eef45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3aa37d80bd4be88566dc19585eef45" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41377 } } peers { permanent_uuid: "f1a6d1bf47274f4288760bb24db5d030" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44993 } } peers { permanent_uuid: "169c7739f5ac4977ad1f4c7c303bf653" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 45721 } }
I20260925 04:54:29.567147 7892 raft_consensus.cc:374] T b982823b08cf42e5998a9888c4728626 P cc3aa37d80bd4be88566dc19585eef45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:29.567595 7892 raft_consensus.cc:729] T b982823b08cf42e5998a9888c4728626 P cc3aa37d80bd4be88566dc19585eef45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc3aa37d80bd4be88566dc19585eef45, State: Initialized, Role: FOLLOWER
I20260925 04:54:29.568285 7892 consensus_queue.cc:260] T b982823b08cf42e5998a9888c4728626 P cc3aa37d80bd4be88566dc19585eef45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3aa37d80bd4be88566dc19585eef45" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41377 } } peers { permanent_uuid: "f1a6d1bf47274f4288760bb24db5d030" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44993 } } peers { permanent_uuid: "169c7739f5ac4977ad1f4c7c303bf653" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 45721 } }
I20260925 04:54:29.568982 7892 ts_tablet_manager.cc:1428] T b982823b08cf42e5998a9888c4728626 P cc3aa37d80bd4be88566dc19585eef45: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260925 04:54:29.582146 7898 raft_consensus.cc:482] T b982823b08cf42e5998a9888c4728626 P f1a6d1bf47274f4288760bb24db5d030 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:54:29.582444 7898 raft_consensus.cc:504] T b982823b08cf42e5998a9888c4728626 P f1a6d1bf47274f4288760bb24db5d030 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3aa37d80bd4be88566dc19585eef45" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41377 } } peers { permanent_uuid: "f1a6d1bf47274f4288760bb24db5d030" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44993 } } peers { permanent_uuid: "169c7739f5ac4977ad1f4c7c303bf653" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 45721 } }
I20260925 04:54:29.584190 7898 leader_election.cc:288] T b982823b08cf42e5998a9888c4728626 P f1a6d1bf47274f4288760bb24db5d030 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc3aa37d80bd4be88566dc19585eef45 (127.6.29.129:41377), 169c7739f5ac4977ad1f4c7c303bf653 (127.6.29.131:45721)
I20260925 04:54:29.593487 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:29.585187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:60133 (local address 127.6.29.131:45721)
0925 04:54:29.585414 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:29.585423 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:29.586471 (+ 1048us) server_negotiation.cc:407] Connection header received
0925 04:54:29.586649 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:29.586654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:29.586727 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:29.586823 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:54:29.587534 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.588711 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.591139 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.592478 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.592632 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:29.592705 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:29.593047 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 04:54:29.593174 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 04:54:29.593794 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:29.584775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:36691 (local address 127.6.29.129:41377)
0925 04:54:29.585008 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:29.585017 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:29.586163 (+ 1146us) server_negotiation.cc:407] Connection header received
0925 04:54:29.586521 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:29.586527 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:29.586635 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:29.586697 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:54:29.587493 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.588288 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.590926 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.592230 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.592389 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:29.592464 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:29.593379 (+ 915us) server_negotiation.cc:299] Negotiation successful
0925 04:54:29.593519 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 04:54:29.593892 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:29.584682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.131:45721 (local address 127.6.29.130:60133)
0925 04:54:29.586349 (+ 1667us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:29.586367 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:29.586501 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:29.586857 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:29.586862 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:29.586889 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:54:29.587282 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:29.587295 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.588859 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.588864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:29.590957 (+ 2093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:29.590971 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.592666 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.592671 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:29.592837 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:29.592860 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 04:54:29.593020 (+ 160us) client_negotiation.cc:240] Negotiation successful
0925 04:54:29.593163 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1514,"mutex_wait_us":3,"thread_start_us":974,"threads_started":1}
I20260925 04:54:29.594508 7922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:29.584359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:41377 (local address 127.6.29.130:36691)
0925 04:54:29.586039 (+ 1680us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:29.586059 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:29.586305 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:29.586918 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:29.586922 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:29.586964 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:54:29.587335 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:29.587344 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.588416 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.588424 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:29.590736 (+ 2312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:29.590751 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.592975 (+ 2224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.592982 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:29.593167 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:29.593200 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 04:54:29.593984 (+ 784us) client_negotiation.cc:240] Negotiation successful
0925 04:54:29.594126 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1517,"mutex_wait_us":75,"thread_start_us":1842,"threads_started":1}
I20260925 04:54:29.594882 7823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b982823b08cf42e5998a9888c4728626" candidate_uuid: "f1a6d1bf47274f4288760bb24db5d030" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "169c7739f5ac4977ad1f4c7c303bf653" is_pre_election: true
I20260925 04:54:29.595242 7550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b982823b08cf42e5998a9888c4728626" candidate_uuid: "f1a6d1bf47274f4288760bb24db5d030" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc3aa37d80bd4be88566dc19585eef45" is_pre_election: true
I20260925 04:54:29.595331 7823 raft_consensus.cc:2454] T b982823b08cf42e5998a9888c4728626 P 169c7739f5ac4977ad1f4c7c303bf653 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a6d1bf47274f4288760bb24db5d030 in term 0.
I20260925 04:54:29.595495 7550 raft_consensus.cc:2454] T b982823b08cf42e5998a9888c4728626 P cc3aa37d80bd4be88566dc19585eef45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a6d1bf47274f4288760bb24db5d030 in term 0.
I20260925 04:54:29.595968 7616 leader_election.cc:302] T b982823b08cf42e5998a9888c4728626 P f1a6d1bf47274f4288760bb24db5d030 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 169c7739f5ac4977ad1f4c7c303bf653, cc3aa37d80bd4be88566dc19585eef45, f1a6d1bf47274f4288760bb24db5d030; no voters:
I20260925 04:54:29.596398 7898 raft_consensus.cc:2781] T b982823b08cf42e5998a9888c4728626 P f1a6d1bf47274f4288760bb24db5d030 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:54:29.596529 7898 raft_consensus.cc:482] T b982823b08cf42e5998a9888c4728626 P f1a6d1bf47274f4288760bb24db5d030 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:54:29.596647 7898 raft_consensus.cc:3037] T b982823b08cf42e5998a9888c4728626 P f1a6d1bf47274f4288760bb24db5d030 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:29.598886 7898 raft_consensus.cc:504] T b982823b08cf42e5998a9888c4728626 P f1a6d1bf47274f4288760bb24db5d030 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3aa37d80bd4be88566dc19585eef45" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41377 } } peers { permanent_uuid: "f1a6d1bf47274f4288760bb24db5d030" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44993 } } peers { permanent_uuid: "169c7739f5ac4977ad1f4c7c303bf653" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 45721 } }
I20260925 04:54:29.599555 7898 leader_election.cc:288] T b982823b08cf42e5998a9888c4728626 P f1a6d1bf47274f4288760bb24db5d030 [CANDIDATE]: Term 1 election: Requested vote from peers cc3aa37d80bd4be88566dc19585eef45 (127.6.29.129:41377), 169c7739f5ac4977ad1f4c7c303bf653 (127.6.29.131:45721)
I20260925 04:54:29.599977 7550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b982823b08cf42e5998a9888c4728626" candidate_uuid: "f1a6d1bf47274f4288760bb24db5d030" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc3aa37d80bd4be88566dc19585eef45"
I20260925 04:54:29.600198 7550 raft_consensus.cc:3037] T b982823b08cf42e5998a9888c4728626 P cc3aa37d80bd4be88566dc19585eef45 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:29.600231 7823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b982823b08cf42e5998a9888c4728626" candidate_uuid: "f1a6d1bf47274f4288760bb24db5d030" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "169c7739f5ac4977ad1f4c7c303bf653"
I20260925 04:54:29.600453 7823 raft_consensus.cc:3037] T b982823b08cf42e5998a9888c4728626 P 169c7739f5ac4977ad1f4c7c303bf653 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:29.603243 7550 raft_consensus.cc:2454] T b982823b08cf42e5998a9888c4728626 P cc3aa37d80bd4be88566dc19585eef45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a6d1bf47274f4288760bb24db5d030 in term 1.
I20260925 04:54:29.603538 7823 raft_consensus.cc:2454] T b982823b08cf42e5998a9888c4728626 P 169c7739f5ac4977ad1f4c7c303bf653 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a6d1bf47274f4288760bb24db5d030 in term 1.
I20260925 04:54:29.603665 7617 leader_election.cc:302] T b982823b08cf42e5998a9888c4728626 P f1a6d1bf47274f4288760bb24db5d030 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cc3aa37d80bd4be88566dc19585eef45, f1a6d1bf47274f4288760bb24db5d030; no voters:
I20260925 04:54:29.604030 7898 raft_consensus.cc:2781] T b982823b08cf42e5998a9888c4728626 P f1a6d1bf47274f4288760bb24db5d030 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:54:29.604561 7898 raft_consensus.cc:686] T b982823b08cf42e5998a9888c4728626 P f1a6d1bf47274f4288760bb24db5d030 [term 1 LEADER]: Becoming Leader. State: Replica: f1a6d1bf47274f4288760bb24db5d030, State: Running, Role: LEADER
I20260925 04:54:29.605156 7898 consensus_queue.cc:237] T b982823b08cf42e5998a9888c4728626 P f1a6d1bf47274f4288760bb24db5d030 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3aa37d80bd4be88566dc19585eef45" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41377 } } peers { permanent_uuid: "f1a6d1bf47274f4288760bb24db5d030" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44993 } } peers { permanent_uuid: "169c7739f5ac4977ad1f4c7c303bf653" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 45721 } }
I20260925 04:54:29.609027 7388 catalog_manager.cc:5495] T b982823b08cf42e5998a9888c4728626 P f1a6d1bf47274f4288760bb24db5d030 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1a6d1bf47274f4288760bb24db5d030 (127.6.29.130). New cstate: current_term: 1 leader_uuid: "f1a6d1bf47274f4288760bb24db5d030" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3aa37d80bd4be88566dc19585eef45" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a6d1bf47274f4288760bb24db5d030" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "169c7739f5ac4977ad1f4c7c303bf653" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 45721 } health_report { overall_health: UNKNOWN } } }
Sep 25 04:54:29 dist-test-slave-xzjr krb5kdc[7343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312067, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:54:29.672698 7931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:29.658438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34434 (local address 127.6.29.190:33387)
0925 04:54:29.659465 (+ 1027us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:29.659470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:29.659496 (+ 26us) server_negotiation.cc:407] Connection header received
0925 04:54:29.659535 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:29.659538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:29.659589 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:29.659702 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:29.661041 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.661931 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.663151 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.663571 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.663705 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:29.667322 (+ 3617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:29.667326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:29.667329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:29.667358 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:29.669967 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:29.670678 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:29.670685 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:29.670689 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:29.670825 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:29.671185 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:29.671193 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:29.671197 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:29.671460 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:29.671606 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:29.672072 (+ 466us) server_negotiation.cc:299] Negotiation successful
0925 04:54:29.672290 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":914,"thread_start_us":767,"threads_started":1}
I20260925 04:54:29.672827 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:29.658394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:33387 (local address 127.0.0.1:34434)
0925 04:54:29.659105 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:29.659132 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:29.659305 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:29.659742 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:29.659749 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:29.660536 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:54:29.660858 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:29.660870 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.662057 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.662064 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:29.663018 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:29.663028 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:29.663843 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:29.663851 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:29.664022 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:29.664791 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:54:29.664811 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:54:29.667114 (+ 2303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:54:29.670124 (+ 3010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:29.670139 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:29.670147 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:29.670561 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:29.670930 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:29.670937 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:29.670941 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:29.671083 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:29.671585 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:54:29.671595 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:54:29.671803 (+ 208us) client_negotiation.cc:769] Sending connection context
0925 04:54:29.671989 (+ 186us) client_negotiation.cc:240] Negotiation successful
0925 04:54:29.672437 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":162,"thread_start_us":352,"threads_started":1}
I20260925 04:54:29.678110 7388 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:34434:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 04:54:29.679370 7388 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:54:29.686419 7525 tablet_service.cc:1467] Processing CreateTablet for tablet d248ea481ceb4590acae25706f31b1bd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d904d3cebfc449d496dc9b65e5c67917]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 04:54:29.686612 7663 tablet_service.cc:1467] Processing CreateTablet for tablet d248ea481ceb4590acae25706f31b1bd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d904d3cebfc449d496dc9b65e5c67917]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 04:54:29.686753 7803 tablet_service.cc:1467] Processing CreateTablet for tablet d248ea481ceb4590acae25706f31b1bd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d904d3cebfc449d496dc9b65e5c67917]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 04:54:29.687052 7525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d248ea481ceb4590acae25706f31b1bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:54:29.687198 7663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d248ea481ceb4590acae25706f31b1bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:54:29.687309 7803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d248ea481ceb4590acae25706f31b1bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:54:29.694415 7894 tablet_bootstrap.cc:492] T d248ea481ceb4590acae25706f31b1bd P f1a6d1bf47274f4288760bb24db5d030: Bootstrap starting.
I20260925 04:54:29.694420 7893 tablet_bootstrap.cc:492] T d248ea481ceb4590acae25706f31b1bd P 169c7739f5ac4977ad1f4c7c303bf653: Bootstrap starting.
I20260925 04:54:29.695331 7892 tablet_bootstrap.cc:492] T d248ea481ceb4590acae25706f31b1bd P cc3aa37d80bd4be88566dc19585eef45: Bootstrap starting.
I20260925 04:54:29.696168 7894 tablet_bootstrap.cc:654] T d248ea481ceb4590acae25706f31b1bd P f1a6d1bf47274f4288760bb24db5d030: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:29.696223 7893 tablet_bootstrap.cc:654] T d248ea481ceb4590acae25706f31b1bd P 169c7739f5ac4977ad1f4c7c303bf653: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:29.697075 7892 tablet_bootstrap.cc:654] T d248ea481ceb4590acae25706f31b1bd P cc3aa37d80bd4be88566dc19585eef45: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:29.706795 7892 tablet_bootstrap.cc:492] T d248ea481ceb4590acae25706f31b1bd P cc3aa37d80bd4be88566dc19585eef45: No bootstrap required, opened a new log
I20260925 04:54:29.706869 7893 tablet_bootstrap.cc:492] T d248ea481ceb4590acae25706f31b1bd P 169c7739f5ac4977ad1f4c7c303bf653: No bootstrap required, opened a new log
I20260925 04:54:29.706995 7892 ts_tablet_manager.cc:1397] T d248ea481ceb4590acae25706f31b1bd P cc3aa37d80bd4be88566dc19585eef45: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260925 04:54:29.707234 7893 ts_tablet_manager.cc:1397] T d248ea481ceb4590acae25706f31b1bd P 169c7739f5ac4977ad1f4c7c303bf653: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260925 04:54:29.707813 7892 raft_consensus.cc:348] T d248ea481ceb4590acae25706f31b1bd P cc3aa37d80bd4be88566dc19585eef45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3aa37d80bd4be88566dc19585eef45" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41377 } } peers { permanent_uuid: "f1a6d1bf47274f4288760bb24db5d030" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44993 } } peers { permanent_uuid: "169c7739f5ac4977ad1f4c7c303bf653" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 45721 } }
I20260925 04:54:29.708092 7892 raft_consensus.cc:374] T d248ea481ceb4590acae25706f31b1bd P cc3aa37d80bd4be88566dc19585eef45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:29.708174 7892 raft_consensus.cc:729] T d248ea481ceb4590acae25706f31b1bd P cc3aa37d80bd4be88566dc19585eef45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc3aa37d80bd4be88566dc19585eef45, State: Initialized, Role: FOLLOWER
I20260925 04:54:29.708472 7892 consensus_queue.cc:260] T d248ea481ceb4590acae25706f31b1bd P cc3aa37d80bd4be88566dc19585eef45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3aa37d80bd4be88566dc19585eef45" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41377 } } peers { permanent_uuid: "f1a6d1bf47274f4288760bb24db5d030" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44993 } } peers { permanent_uuid: "169c7739f5ac4977ad1f4c7c303bf653" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 45721 } }
I20260925 04:54:29.708714 7894 tablet_bootstrap.cc:492] T d248ea481ceb4590acae25706f31b1bd P f1a6d1bf47274f4288760bb24db5d030: No bootstrap required, opened a new log
I20260925 04:54:29.708901 7894 ts_tablet_manager.cc:1397] T d248ea481ceb4590acae25706f31b1bd P f1a6d1bf47274f4288760bb24db5d030: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20260925 04:54:29.708752 7893 raft_consensus.cc:348] T d248ea481ceb4590acae25706f31b1bd P 169c7739f5ac4977ad1f4c7c303bf653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3aa37d80bd4be88566dc19585eef45" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41377 } } peers { permanent_uuid: "f1a6d1bf47274f4288760bb24db5d030" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44993 } } peers { permanent_uuid: "169c7739f5ac4977ad1f4c7c303bf653" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 45721 } }
I20260925 04:54:29.709141 7893 raft_consensus.cc:374] T d248ea481ceb4590acae25706f31b1bd P 169c7739f5ac4977ad1f4c7c303bf653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:29.708948 7899 tablet_replica.cc:439] T d248ea481ceb4590acae25706f31b1bd P cc3aa37d80bd4be88566dc19585eef45: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3aa37d80bd4be88566dc19585eef45" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41377 } } peers { permanent_uuid: "f1a6d1bf47274f4288760bb24db5d030" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44993 } } peers { permanent_uuid: "169c7739f5ac4977ad1f4c7c303bf653" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 45721 } } }
I20260925 04:54:29.709328 7899 tablet_replica.cc:442] T d248ea481ceb4590acae25706f31b1bd P cc3aa37d80bd4be88566dc19585eef45: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:54:29.709239 7893 raft_consensus.cc:729] T d248ea481ceb4590acae25706f31b1bd P 169c7739f5ac4977ad1f4c7c303bf653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 169c7739f5ac4977ad1f4c7c303bf653, State: Initialized, Role: FOLLOWER
I20260925 04:54:29.709662 7894 raft_consensus.cc:348] T d248ea481ceb4590acae25706f31b1bd P f1a6d1bf47274f4288760bb24db5d030 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3aa37d80bd4be88566dc19585eef45" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41377 } } peers { permanent_uuid: "f1a6d1bf47274f4288760bb24db5d030" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44993 } } peers { permanent_uuid: "169c7739f5ac4977ad1f4c7c303bf653" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 45721 } }
I20260925 04:54:29.709911 7894 raft_consensus.cc:374] T d248ea481ceb4590acae25706f31b1bd P f1a6d1bf47274f4288760bb24db5d030 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:29.709739 7893 consensus_queue.cc:260] T d248ea481ceb4590acae25706f31b1bd P 169c7739f5ac4977ad1f4c7c303bf653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3aa37d80bd4be88566dc19585eef45" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41377 } } peers { permanent_uuid: "f1a6d1bf47274f4288760bb24db5d030" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44993 } } peers { permanent_uuid: "169c7739f5ac4977ad1f4c7c303bf653" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 45721 } }
I20260925 04:54:29.709992 7894 raft_consensus.cc:729] T d248ea481ceb4590acae25706f31b1bd P f1a6d1bf47274f4288760bb24db5d030 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a6d1bf47274f4288760bb24db5d030, State: Initialized, Role: FOLLOWER
I20260925 04:54:29.710227 7894 consensus_queue.cc:260] T d248ea481ceb4590acae25706f31b1bd P f1a6d1bf47274f4288760bb24db5d030 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3aa37d80bd4be88566dc19585eef45" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41377 } } peers { permanent_uuid: "f1a6d1bf47274f4288760bb24db5d030" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44993 } } peers { permanent_uuid: "169c7739f5ac4977ad1f4c7c303bf653" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 45721 } }
I20260925 04:54:29.710296 7913 tablet_replica.cc:439] T d248ea481ceb4590acae25706f31b1bd P 169c7739f5ac4977ad1f4c7c303bf653: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3aa37d80bd4be88566dc19585eef45" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41377 } } peers { permanent_uuid: "f1a6d1bf47274f4288760bb24db5d030" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44993 } } peers { permanent_uuid: "169c7739f5ac4977ad1f4c7c303bf653" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 45721 } } }
I20260925 04:54:29.710641 7913 tablet_replica.cc:442] T d248ea481ceb4590acae25706f31b1bd P 169c7739f5ac4977ad1f4c7c303bf653: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:54:29.710667 7892 ts_tablet_manager.cc:1428] T d248ea481ceb4590acae25706f31b1bd P cc3aa37d80bd4be88566dc19585eef45: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260925 04:54:29.710791 7894 ts_tablet_manager.cc:1428] T d248ea481ceb4590acae25706f31b1bd P f1a6d1bf47274f4288760bb24db5d030: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 04:54:29.711445 7898 tablet_replica.cc:439] T d248ea481ceb4590acae25706f31b1bd P f1a6d1bf47274f4288760bb24db5d030: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3aa37d80bd4be88566dc19585eef45" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41377 } } peers { permanent_uuid: "f1a6d1bf47274f4288760bb24db5d030" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44993 } } peers { permanent_uuid: "169c7739f5ac4977ad1f4c7c303bf653" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 45721 } } }
I20260925 04:54:29.711719 7898 tablet_replica.cc:442] T d248ea481ceb4590acae25706f31b1bd P f1a6d1bf47274f4288760bb24db5d030: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:54:29.714064 7893 ts_tablet_manager.cc:1428] T d248ea481ceb4590acae25706f31b1bd P 169c7739f5ac4977ad1f4c7c303bf653: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20260925 04:54:29.997355 7899 raft_consensus.cc:482] T d248ea481ceb4590acae25706f31b1bd P cc3aa37d80bd4be88566dc19585eef45 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:54:29.997592 7899 raft_consensus.cc:504] T d248ea481ceb4590acae25706f31b1bd P cc3aa37d80bd4be88566dc19585eef45 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3aa37d80bd4be88566dc19585eef45" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41377 } } peers { permanent_uuid: "f1a6d1bf47274f4288760bb24db5d030" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44993 } } peers { permanent_uuid: "169c7739f5ac4977ad1f4c7c303bf653" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 45721 } }
I20260925 04:54:29.999059 7899 leader_election.cc:288] T d248ea481ceb4590acae25706f31b1bd P cc3aa37d80bd4be88566dc19585eef45 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1a6d1bf47274f4288760bb24db5d030 (127.6.29.130:44993), 169c7739f5ac4977ad1f4c7c303bf653 (127.6.29.131:45721)
I20260925 04:54:30.006057 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:29.999597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:56317 (local address 127.6.29.131:45721)
0925 04:54:29.999836 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:29.999845 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:30.000107 (+ 262us) server_negotiation.cc:407] Connection header received
0925 04:54:30.000429 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:30.000438 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:30.000524 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:30.000595 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:54:30.001217 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:30.002038 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:30.004327 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:30.005057 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:30.005228 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:30.005316 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:30.005674 (+ 358us) server_negotiation.cc:299] Negotiation successful
0925 04:54:30.005800 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 04:54:30.006192 7936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:29.999515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:44993 (local address 127.6.29.129:40095)
0925 04:54:30.000502 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:30.000519 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:30.000640 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:30.000966 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:30.000970 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:30.000993 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:54:30.001418 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:30.001428 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:30.002641 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:30.002649 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:30.004100 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:30.004112 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:30.005309 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:30.005314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:30.005460 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:30.005502 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 04:54:30.005573 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 04:54:30.005724 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":860,"mutex_wait_us":5,"spinlock_wait_cycles":19712,"thread_start_us":707,"threads_started":1}
I20260925 04:54:30.006230 7935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:29.999472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.131:45721 (local address 127.6.29.129:56317)
0925 04:54:30.000013 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:30.000032 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:30.000229 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:30.000755 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:30.000758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:30.000778 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:54:30.001063 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:30.001070 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:30.002171 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:30.002177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:30.004158 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:30.004192 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:30.005275 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:30.005283 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:30.005457 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:30.005526 (+ 69us) client_negotiation.cc:769] Sending connection context
0925 04:54:30.005632 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 04:54:30.005762 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":120,"spinlock_wait_cycles":5504,"thread_start_us":399,"threads_started":1}
I20260925 04:54:30.006078 7937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:29.999587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:40095 (local address 127.6.29.130:44993)
0925 04:54:30.000689 (+ 1102us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:30.000698 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:30.000717 (+ 19us) server_negotiation.cc:407] Connection header received
0925 04:54:30.000786 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:30.000789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:30.000841 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:30.000948 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:54:30.001586 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:30.002501 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:30.004250 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:30.005059 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:30.005245 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:30.005316 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:30.005625 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 04:54:30.005760 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":984,"thread_start_us":855,"threads_started":1}
I20260925 04:54:30.006870 7823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d248ea481ceb4590acae25706f31b1bd" candidate_uuid: "cc3aa37d80bd4be88566dc19585eef45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "169c7739f5ac4977ad1f4c7c303bf653" is_pre_election: true
I20260925 04:54:30.007097 7823 raft_consensus.cc:2454] T d248ea481ceb4590acae25706f31b1bd P 169c7739f5ac4977ad1f4c7c303bf653 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc3aa37d80bd4be88566dc19585eef45 in term 0.
I20260925 04:54:30.007093 7684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d248ea481ceb4590acae25706f31b1bd" candidate_uuid: "cc3aa37d80bd4be88566dc19585eef45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a6d1bf47274f4288760bb24db5d030" is_pre_election: true
I20260925 04:54:30.007341 7684 raft_consensus.cc:2454] T d248ea481ceb4590acae25706f31b1bd P f1a6d1bf47274f4288760bb24db5d030 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc3aa37d80bd4be88566dc19585eef45 in term 0.
I20260925 04:54:30.007593 7476 leader_election.cc:302] T d248ea481ceb4590acae25706f31b1bd P cc3aa37d80bd4be88566dc19585eef45 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 169c7739f5ac4977ad1f4c7c303bf653, cc3aa37d80bd4be88566dc19585eef45; no voters:
I20260925 04:54:30.008071 7899 raft_consensus.cc:2781] T d248ea481ceb4590acae25706f31b1bd P cc3aa37d80bd4be88566dc19585eef45 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:54:30.008208 7899 raft_consensus.cc:482] T d248ea481ceb4590acae25706f31b1bd P cc3aa37d80bd4be88566dc19585eef45 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:54:30.008312 7899 raft_consensus.cc:3037] T d248ea481ceb4590acae25706f31b1bd P cc3aa37d80bd4be88566dc19585eef45 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:30.010366 7899 raft_consensus.cc:504] T d248ea481ceb4590acae25706f31b1bd P cc3aa37d80bd4be88566dc19585eef45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3aa37d80bd4be88566dc19585eef45" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41377 } } peers { permanent_uuid: "f1a6d1bf47274f4288760bb24db5d030" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44993 } } peers { permanent_uuid: "169c7739f5ac4977ad1f4c7c303bf653" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 45721 } }
I20260925 04:54:30.010957 7899 leader_election.cc:288] T d248ea481ceb4590acae25706f31b1bd P cc3aa37d80bd4be88566dc19585eef45 [CANDIDATE]: Term 1 election: Requested vote from peers f1a6d1bf47274f4288760bb24db5d030 (127.6.29.130:44993), 169c7739f5ac4977ad1f4c7c303bf653 (127.6.29.131:45721)
I20260925 04:54:30.011459 7823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d248ea481ceb4590acae25706f31b1bd" candidate_uuid: "cc3aa37d80bd4be88566dc19585eef45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "169c7739f5ac4977ad1f4c7c303bf653"
I20260925 04:54:30.011475 7684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d248ea481ceb4590acae25706f31b1bd" candidate_uuid: "cc3aa37d80bd4be88566dc19585eef45" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a6d1bf47274f4288760bb24db5d030"
I20260925 04:54:30.011672 7684 raft_consensus.cc:3037] T d248ea481ceb4590acae25706f31b1bd P f1a6d1bf47274f4288760bb24db5d030 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:30.011693 7823 raft_consensus.cc:3037] T d248ea481ceb4590acae25706f31b1bd P 169c7739f5ac4977ad1f4c7c303bf653 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:30.014544 7684 raft_consensus.cc:2454] T d248ea481ceb4590acae25706f31b1bd P f1a6d1bf47274f4288760bb24db5d030 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc3aa37d80bd4be88566dc19585eef45 in term 1.
I20260925 04:54:30.014532 7823 raft_consensus.cc:2454] T d248ea481ceb4590acae25706f31b1bd P 169c7739f5ac4977ad1f4c7c303bf653 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc3aa37d80bd4be88566dc19585eef45 in term 1.
I20260925 04:54:30.014976 7478 leader_election.cc:302] T d248ea481ceb4590acae25706f31b1bd P cc3aa37d80bd4be88566dc19585eef45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cc3aa37d80bd4be88566dc19585eef45, f1a6d1bf47274f4288760bb24db5d030; no voters:
I20260925 04:54:30.015296 7899 raft_consensus.cc:2781] T d248ea481ceb4590acae25706f31b1bd P cc3aa37d80bd4be88566dc19585eef45 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:54:30.015692 7899 raft_consensus.cc:686] T d248ea481ceb4590acae25706f31b1bd P cc3aa37d80bd4be88566dc19585eef45 [term 1 LEADER]: Becoming Leader. State: Replica: cc3aa37d80bd4be88566dc19585eef45, State: Running, Role: LEADER
I20260925 04:54:30.016145 7899 consensus_queue.cc:237] T d248ea481ceb4590acae25706f31b1bd P cc3aa37d80bd4be88566dc19585eef45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3aa37d80bd4be88566dc19585eef45" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41377 } } peers { permanent_uuid: "f1a6d1bf47274f4288760bb24db5d030" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44993 } } peers { permanent_uuid: "169c7739f5ac4977ad1f4c7c303bf653" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 45721 } }
I20260925 04:54:30.018059 7938 tablet_replica.cc:439] T d248ea481ceb4590acae25706f31b1bd P cc3aa37d80bd4be88566dc19585eef45: TxnStatusTablet state changed. Reason: New leader cc3aa37d80bd4be88566dc19585eef45. Latest consensus state: current_term: 1 leader_uuid: "cc3aa37d80bd4be88566dc19585eef45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3aa37d80bd4be88566dc19585eef45" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41377 } } peers { permanent_uuid: "f1a6d1bf47274f4288760bb24db5d030" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44993 } } peers { permanent_uuid: "169c7739f5ac4977ad1f4c7c303bf653" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 45721 } } }
I20260925 04:54:30.018399 7938 tablet_replica.cc:442] T d248ea481ceb4590acae25706f31b1bd P cc3aa37d80bd4be88566dc19585eef45: This TxnStatusTablet replica's current role is: LEADER
I20260925 04:54:30.019117 7940 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 04:54:30.019325 7940 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 04:54:30.019920 7386 catalog_manager.cc:5495] T d248ea481ceb4590acae25706f31b1bd P cc3aa37d80bd4be88566dc19585eef45 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc3aa37d80bd4be88566dc19585eef45 (127.6.29.129). New cstate: current_term: 1 leader_uuid: "cc3aa37d80bd4be88566dc19585eef45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3aa37d80bd4be88566dc19585eef45" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a6d1bf47274f4288760bb24db5d030" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "169c7739f5ac4977ad1f4c7c303bf653" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 45721 } health_report { overall_health: UNKNOWN } } }
I20260925 04:54:30.076860 7898 consensus_queue.cc:1035] T b982823b08cf42e5998a9888c4728626 P f1a6d1bf47274f4288760bb24db5d030 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc3aa37d80bd4be88566dc19585eef45" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:54:30.078953 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:30.071315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:41377 (local address 127.0.0.1:36852)
0925 04:54:30.071586 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:30.071607 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:30.071846 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:30.072386 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:30.072391 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:30.072405 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:54:30.072587 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:30.072595 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:30.073702 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:30.073711 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:30.075172 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:30.075190 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:30.076154 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:30.076160 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:30.076347 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:30.076441 (+ 94us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:54:30.077749 (+ 1308us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:54:30.077772 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 04:54:30.077866 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 04:54:30.078026 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260925 04:54:30.079584 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:30.071484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36852 (local address 127.6.29.129:41377)
0925 04:54:30.071673 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:30.071678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:30.071700 (+ 22us) server_negotiation.cc:407] Connection header received
0925 04:54:30.072100 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:30.072104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:30.072200 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:30.072355 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:54:30.072728 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:30.073553 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:30.075333 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:30.075857 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:30.076008 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:30.076669 (+ 661us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:54:30.077552 (+ 883us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:54:30.077690 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:30.078261 (+ 571us) server_negotiation.cc:299] Negotiation successful
0925 04:54:30.078342 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 04:54:30.085111 7898 consensus_queue.cc:1035] T b982823b08cf42e5998a9888c4728626 P f1a6d1bf47274f4288760bb24db5d030 [LEADER]: Connected to new peer: Peer: permanent_uuid: "169c7739f5ac4977ad1f4c7c303bf653" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 45721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:54:30.087782 7822 raft_consensus.cc:1261] T d248ea481ceb4590acae25706f31b1bd P 169c7739f5ac4977ad1f4c7c303bf653 [term 1 FOLLOWER]: Refusing update from remote peer cc3aa37d80bd4be88566dc19585eef45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:54:30.088373 7684 raft_consensus.cc:1261] T d248ea481ceb4590acae25706f31b1bd P f1a6d1bf47274f4288760bb24db5d030 [term 1 FOLLOWER]: Refusing update from remote peer cc3aa37d80bd4be88566dc19585eef45: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:54:30.088737 7938 consensus_queue.cc:1035] T d248ea481ceb4590acae25706f31b1bd P cc3aa37d80bd4be88566dc19585eef45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "169c7739f5ac4977ad1f4c7c303bf653" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 45721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:54:30.089540 7938 consensus_queue.cc:1035] T d248ea481ceb4590acae25706f31b1bd P cc3aa37d80bd4be88566dc19585eef45 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a6d1bf47274f4288760bb24db5d030" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:54:30.094648 7913 tablet_replica.cc:439] T d248ea481ceb4590acae25706f31b1bd P 169c7739f5ac4977ad1f4c7c303bf653: TxnStatusTablet state changed. Reason: New leader cc3aa37d80bd4be88566dc19585eef45. Latest consensus state: current_term: 1 leader_uuid: "cc3aa37d80bd4be88566dc19585eef45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3aa37d80bd4be88566dc19585eef45" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41377 } } peers { permanent_uuid: "f1a6d1bf47274f4288760bb24db5d030" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44993 } } peers { permanent_uuid: "169c7739f5ac4977ad1f4c7c303bf653" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 45721 } } }
I20260925 04:54:30.094992 7913 tablet_replica.cc:442] T d248ea481ceb4590acae25706f31b1bd P 169c7739f5ac4977ad1f4c7c303bf653: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:54:30.109225 7913 tablet_replica.cc:439] T d248ea481ceb4590acae25706f31b1bd P 169c7739f5ac4977ad1f4c7c303bf653: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cc3aa37d80bd4be88566dc19585eef45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3aa37d80bd4be88566dc19585eef45" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41377 } } peers { permanent_uuid: "f1a6d1bf47274f4288760bb24db5d030" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44993 } } peers { permanent_uuid: "169c7739f5ac4977ad1f4c7c303bf653" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 45721 } } }
I20260925 04:54:30.109630 7913 tablet_replica.cc:442] T d248ea481ceb4590acae25706f31b1bd P 169c7739f5ac4977ad1f4c7c303bf653: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:54:30.116123 7899 tablet_replica.cc:439] T d248ea481ceb4590acae25706f31b1bd P cc3aa37d80bd4be88566dc19585eef45: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cc3aa37d80bd4be88566dc19585eef45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3aa37d80bd4be88566dc19585eef45" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41377 } } peers { permanent_uuid: "f1a6d1bf47274f4288760bb24db5d030" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44993 } } peers { permanent_uuid: "169c7739f5ac4977ad1f4c7c303bf653" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 45721 } } }
I20260925 04:54:30.116743 7899 tablet_replica.cc:442] T d248ea481ceb4590acae25706f31b1bd P cc3aa37d80bd4be88566dc19585eef45: This TxnStatusTablet replica's current role is: LEADER
I20260925 04:54:30.138150 7898 tablet_replica.cc:439] T d248ea481ceb4590acae25706f31b1bd P f1a6d1bf47274f4288760bb24db5d030: TxnStatusTablet state changed. Reason: New leader cc3aa37d80bd4be88566dc19585eef45. Latest consensus state: current_term: 1 leader_uuid: "cc3aa37d80bd4be88566dc19585eef45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3aa37d80bd4be88566dc19585eef45" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41377 } } peers { permanent_uuid: "f1a6d1bf47274f4288760bb24db5d030" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44993 } } peers { permanent_uuid: "169c7739f5ac4977ad1f4c7c303bf653" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 45721 } } }
I20260925 04:54:30.138505 7898 tablet_replica.cc:442] T d248ea481ceb4590acae25706f31b1bd P f1a6d1bf47274f4288760bb24db5d030: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:54:30.140669 7899 tablet_replica.cc:439] T d248ea481ceb4590acae25706f31b1bd P cc3aa37d80bd4be88566dc19585eef45: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cc3aa37d80bd4be88566dc19585eef45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3aa37d80bd4be88566dc19585eef45" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41377 } } peers { permanent_uuid: "f1a6d1bf47274f4288760bb24db5d030" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44993 } } peers { permanent_uuid: "169c7739f5ac4977ad1f4c7c303bf653" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 45721 } } }
I20260925 04:54:30.141005 7899 tablet_replica.cc:442] T d248ea481ceb4590acae25706f31b1bd P cc3aa37d80bd4be88566dc19585eef45: This TxnStatusTablet replica's current role is: LEADER
I20260925 04:54:30.148932 7898 tablet_replica.cc:439] T d248ea481ceb4590acae25706f31b1bd P f1a6d1bf47274f4288760bb24db5d030: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cc3aa37d80bd4be88566dc19585eef45" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc3aa37d80bd4be88566dc19585eef45" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41377 } } peers { permanent_uuid: "f1a6d1bf47274f4288760bb24db5d030" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44993 } } peers { permanent_uuid: "169c7739f5ac4977ad1f4c7c303bf653" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 45721 } } }
I20260925 04:54:30.149258 7937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:30.119525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44250 (local address 127.6.29.130:44993)
0925 04:54:30.119744 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:30.119753 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:30.141373 (+ 21620us) server_negotiation.cc:407] Connection header received
0925 04:54:30.141901 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:30.141908 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:30.142016 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:30.142149 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:54:30.142923 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:30.144104 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:30.146265 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:30.146930 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:30.147027 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:30.147670 (+ 643us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:54:30.148472 (+ 802us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:54:30.148594 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:30.148837 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 04:54:30.148952 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 04:54:30.149266 7898 tablet_replica.cc:442] T d248ea481ceb4590acae25706f31b1bd P f1a6d1bf47274f4288760bb24db5d030: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:54:30.167088 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:30.161565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:33129 (local address 127.6.29.129:41377)
0925 04:54:30.161837 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:30.161843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:30.162026 (+ 183us) server_negotiation.cc:407] Connection header received
0925 04:54:30.162242 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:30.162246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:30.162321 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:30.162408 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:54:30.162936 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:30.163718 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:30.165210 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:30.165700 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:30.165858 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:30.166268 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:54:30.166404 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:54:30.166514 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:30.166662 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 04:54:30.166741 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260925 04:54:30.167133 7961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:30.161289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:41377 (local address 127.6.29.130:33129)
0925 04:54:30.161921 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:30.161941 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:30.162115 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:30.162447 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:30.162452 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:30.162464 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:54:30.162737 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:30.162748 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:30.163964 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:30.163972 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:30.165058 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:30.165071 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:30.165876 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:30.165882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:30.166034 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:30.166087 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:54:30.166515 (+ 428us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:54:30.166528 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 04:54:30.166628 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 04:54:30.166762 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":302,"threads_started":1}
I20260925 04:54:30.206342 7937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:30.199500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:40179 (local address 127.6.29.130:44993)
0925 04:54:30.199673 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:30.199679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:30.200431 (+ 752us) server_negotiation.cc:407] Connection header received
0925 04:54:30.200656 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:30.200661 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:30.200737 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:30.200826 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:54:30.201507 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:30.202659 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:30.204378 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:30.204951 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:30.205103 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:30.205627 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:54:30.205723 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:54:30.205830 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:30.205955 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 04:54:30.206049 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 04:54:30.206533 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:30.199535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:44993 (local address 127.6.29.129:40179)
0925 04:54:30.200331 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:30.200361 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:30.200505 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:30.200852 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:30.200856 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:30.200868 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:54:30.201306 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:30.201318 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:30.202794 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:30.202797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:30.204204 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:30.204217 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:30.205143 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:30.205151 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:30.205358 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:30.205436 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:54:30.205833 (+ 397us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:54:30.205845 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 04:54:30.205945 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 04:54:30.206092 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":463,"threads_started":1}
I20260925 04:54:30.262718 7386 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:54:30.262986 7386 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:54:30.266705 7386 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b60b88bcac904dd584ad8b377a235dc1: Sending DeleteTablet for 3 replicas of tablet b982823b08cf42e5998a9888c4728626
I20260925 04:54:30.267597 7525 tablet_service.cc:1514] Processing DeleteTablet for tablet b982823b08cf42e5998a9888c4728626 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:54:30 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:58382
I20260925 04:54:30.267915 7803 tablet_service.cc:1514] Processing DeleteTablet for tablet b982823b08cf42e5998a9888c4728626 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:54:30 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:51578
I20260925 04:54:30.268311 7663 tablet_service.cc:1514] Processing DeleteTablet for tablet b982823b08cf42e5998a9888c4728626 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:54:30 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:34990
I20260925 04:54:30.289207 7966 tablet_replica.cc:330] T b982823b08cf42e5998a9888c4728626 P cc3aa37d80bd4be88566dc19585eef45: stopping tablet replica
I20260925 04:54:30.289623 7966 raft_consensus.cc:2229] T b982823b08cf42e5998a9888c4728626 P cc3aa37d80bd4be88566dc19585eef45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:54:30.290004 7966 raft_consensus.cc:2258] T b982823b08cf42e5998a9888c4728626 P cc3aa37d80bd4be88566dc19585eef45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:54:30.291633 7966 ts_tablet_manager.cc:1905] T b982823b08cf42e5998a9888c4728626 P cc3aa37d80bd4be88566dc19585eef45: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:54:30.295061 7968 tablet_replica.cc:330] T b982823b08cf42e5998a9888c4728626 P f1a6d1bf47274f4288760bb24db5d030: stopping tablet replica
I20260925 04:54:30.295290 7968 raft_consensus.cc:2229] T b982823b08cf42e5998a9888c4728626 P f1a6d1bf47274f4288760bb24db5d030 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:54:30.295921 7968 raft_consensus.cc:2258] T b982823b08cf42e5998a9888c4728626 P f1a6d1bf47274f4288760bb24db5d030 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:54:30.297096 7968 ts_tablet_manager.cc:1905] T b982823b08cf42e5998a9888c4728626 P f1a6d1bf47274f4288760bb24db5d030: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:54:30.300328 7967 tablet_replica.cc:330] T b982823b08cf42e5998a9888c4728626 P 169c7739f5ac4977ad1f4c7c303bf653: stopping tablet replica
I20260925 04:54:30.300596 7967 raft_consensus.cc:2229] T b982823b08cf42e5998a9888c4728626 P 169c7739f5ac4977ad1f4c7c303bf653 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:54:30.300858 7967 raft_consensus.cc:2258] T b982823b08cf42e5998a9888c4728626 P 169c7739f5ac4977ad1f4c7c303bf653 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:54:30.301954 7967 ts_tablet_manager.cc:1905] T b982823b08cf42e5998a9888c4728626 P 169c7739f5ac4977ad1f4c7c303bf653: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:54:30.303686 7966 ts_tablet_manager.cc:1918] T b982823b08cf42e5998a9888c4728626 P cc3aa37d80bd4be88566dc19585eef45: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:54:30.303849 7966 log.cc:1196] T b982823b08cf42e5998a9888c4728626 P cc3aa37d80bd4be88566dc19585eef45: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-0/wal/wals/b982823b08cf42e5998a9888c4728626
I20260925 04:54:30.304351 7966 ts_tablet_manager.cc:1939] T b982823b08cf42e5998a9888c4728626 P cc3aa37d80bd4be88566dc19585eef45: Deleting consensus metadata
I20260925 04:54:30.305465 7374 catalog_manager.cc:4841] TS cc3aa37d80bd4be88566dc19585eef45 (127.6.29.129:41377): tablet b982823b08cf42e5998a9888c4728626 (table test-table [id=e711c99d71f249f983377afd45ec7a06]) successfully deleted
I20260925 04:54:30.312896 7967 ts_tablet_manager.cc:1918] T b982823b08cf42e5998a9888c4728626 P 169c7739f5ac4977ad1f4c7c303bf653: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:54:30.313076 7967 log.cc:1196] T b982823b08cf42e5998a9888c4728626 P 169c7739f5ac4977ad1f4c7c303bf653: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-2/wal/wals/b982823b08cf42e5998a9888c4728626
I20260925 04:54:30.313553 7967 ts_tablet_manager.cc:1939] T b982823b08cf42e5998a9888c4728626 P 169c7739f5ac4977ad1f4c7c303bf653: Deleting consensus metadata
I20260925 04:54:30.314845 7373 catalog_manager.cc:4841] TS 169c7739f5ac4977ad1f4c7c303bf653 (127.6.29.131:45721): tablet b982823b08cf42e5998a9888c4728626 (table test-table [id=e711c99d71f249f983377afd45ec7a06]) successfully deleted
I20260925 04:54:30.314447 7968 ts_tablet_manager.cc:1918] T b982823b08cf42e5998a9888c4728626 P f1a6d1bf47274f4288760bb24db5d030: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:54:30.315196 7968 log.cc:1196] T b982823b08cf42e5998a9888c4728626 P f1a6d1bf47274f4288760bb24db5d030: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790312053666503-6262-0/minicluster-data/ts-1/wal/wals/b982823b08cf42e5998a9888c4728626
I20260925 04:54:30.315835 7968 ts_tablet_manager.cc:1939] T b982823b08cf42e5998a9888c4728626 P f1a6d1bf47274f4288760bb24db5d030: Deleting consensus metadata
I20260925 04:54:30.316993 7375 catalog_manager.cc:4841] TS f1a6d1bf47274f4288760bb24db5d030 (127.6.29.130:44993): tablet b982823b08cf42e5998a9888c4728626 (table test-table [id=e711c99d71f249f983377afd45ec7a06]) successfully deleted
Sep 25 04:54:30 dist-test-slave-xzjr krb5kdc[7343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312069, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.29.129@KRBTEST.COM
I20260925 04:54:30.328531 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:30.294483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36858 (local address 127.6.29.129:41377)
0925 04:54:30.294689 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:30.294695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:30.304400 (+ 9705us) server_negotiation.cc:407] Connection header received
0925 04:54:30.304722 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:30.304727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:30.304840 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:30.304923 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:30.306777 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:30.307933 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:30.309760 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:30.310395 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:30.310493 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:30.320799 (+ 10306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:30.320813 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:30.320833 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:30.320882 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:30.325008 (+ 4126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:30.325958 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:30.325969 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:30.325975 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:30.326144 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:30.326569 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:30.326574 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:30.326579 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:30.327013 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:30.327316 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:30.327742 (+ 426us) server_negotiation.cc:299] Negotiation successful
0925 04:54:30.327987 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260925 04:54:30.329737 7590 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36858
I20260925 04:54:30.364408 7973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:30.333546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34436 (local address 127.6.29.190:33387)
0925 04:54:30.348235 (+ 14689us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:30.348242 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:30.348266 (+ 24us) server_negotiation.cc:407] Connection header received
0925 04:54:30.348351 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:30.348358 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:30.348464 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:30.348599 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:30.350385 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:30.351598 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:30.353485 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:30.354170 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:30.354330 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:30.357326 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:30.357355 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:30.357361 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:30.357426 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:30.361258 (+ 3832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:30.362169 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:30.362175 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:30.362179 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:30.362339 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:30.362749 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:30.362754 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:30.362758 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:30.363113 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:30.363463 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:30.363728 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 04:54:30.363984 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14543,"thread_start_us":14404,"threads_started":1}
W20260925 04:54:30.365094 7386 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34436
I20260925 04:54:30.367262 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 7459
I20260925 04:54:30.385784 7891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:30.368213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36868 (local address 127.6.29.129:41377)
0925 04:54:30.368409 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:30.368415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:30.369134 (+ 719us) server_negotiation.cc:407] Connection header received
0925 04:54:30.369395 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:30.369402 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:30.369481 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:30.369551 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:30.371194 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:30.372496 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:30.374406 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:30.375149 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:30.375298 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:30.378114 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:30.378120 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:30.378125 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:30.378165 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:30.382019 (+ 3854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:30.382805 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:30.382811 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:30.382814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:30.382965 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:30.383393 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:30.383398 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:30.383402 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:30.383826 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:30.384071 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:30.385168 (+ 1097us) server_negotiation.cc:299] Negotiation successful
0925 04:54:30.385400 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260925 04:54:30.386395 7590 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36868
W20260925 04:54:30.386967 6262 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 7459: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 04:54:30.387172 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 7459
I20260925 04:54:30.407600 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 7599
Sep 25 04:54:30 dist-test-slave-xzjr krb5kdc[7343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312069, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.29.130@KRBTEST.COM
I20260925 04:54:30.423240 7937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:30.408479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44262 (local address 127.6.29.130:44993)
0925 04:54:30.408659 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:30.408665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:30.408686 (+ 21us) server_negotiation.cc:407] Connection header received
0925 04:54:30.408806 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:30.408810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:30.408884 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:30.409000 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:30.410134 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:30.411008 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:30.412371 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:30.412805 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:30.412948 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:30.417094 (+ 4146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:30.417107 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:30.417124 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:30.417160 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:30.420396 (+ 3236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:30.421118 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:30.421125 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:30.421128 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:30.421259 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:30.421647 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:30.421653 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:30.421656 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:30.422065 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:30.422373 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:30.422644 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 04:54:30.422879 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260925 04:54:30.424104 7729 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44262
W20260925 04:54:30.424796 6262 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 7599: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 04:54:30.424957 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 7599
I20260925 04:54:30.449261 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 7738
Sep 25 04:54:30 dist-test-slave-xzjr krb5kdc[7343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312069, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.29.131@KRBTEST.COM
I20260925 04:54:30.465013 7889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:30.450046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57366 (local address 127.6.29.131:45721)
0925 04:54:30.450221 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:30.450227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:30.450248 (+ 21us) server_negotiation.cc:407] Connection header received
0925 04:54:30.450400 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:30.450406 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:30.450481 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:30.450608 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:30.451857 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:30.452953 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:30.454221 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:30.454640 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:30.454739 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:30.459021 (+ 4282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:30.459036 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:30.459053 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:30.459086 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:30.462107 (+ 3021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:30.462762 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:30.462770 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:30.462772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:30.462905 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:30.463254 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:30.463259 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:30.463261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:30.463804 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:30.464080 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:30.464398 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 04:54:30.464651 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260925 04:54:30.465755 7868 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57366
W20260925 04:54:30.466424 6262 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 7738: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 04:54:30.466596 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 7738
I20260925 04:54:30.487586 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 7356
I20260925 04:54:30.488766 7428 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:54:30.603639 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 7356
2026-09-25T04:54:30Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4915 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:54:30 dist-test-slave-xzjr krb5kdc[7978](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:54:30 dist-test-slave-xzjr krb5kdc[7978](info): set up 2 sockets
Sep 25 04:54:30 dist-test-slave-xzjr krb5kdc[7978](info): commencing operation
krb5kdc: starting...
W20260925 04:54:31.684544 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 04:54:31 dist-test-slave-xzjr krb5kdc[7978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:54:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:54:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.29.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
I20260925 04:54:31.871362 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:34435
--webserver_interface=127.6.29.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:34565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:34435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:54:32.192435 7991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:54:32.192764 7991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:54:32.192931 7991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:54:32.200306 7991 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:54:32.200428 7991 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 04:54:32.200506 7991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:54:32.200608 7991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:54:32.200692 7991 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:54:32.200806 7991 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:54:32.211042 7991 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:34565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:34435
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:34435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:54:32.212505 7991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:54:32.213975 7991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:54:32.229672 7996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:32.229609 7997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:32.229641 7999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:54:32.230845 7991 server_base.cc:1034] running on GCE node
I20260925 04:54:32.231639 7991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:54:32.233214 7991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:54:32.234445 7991 hybrid_clock.cc:648] HybridClock initialized: now 1790312072234379 us; error 41 us; skew 500 ppm
I20260925 04:54:32.234830 7991 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:54:32 dist-test-slave-xzjr krb5kdc[7978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312072, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:54:32.238375 7991 init.cc:377] Logged in from keytab as kudu/127.6.29.190@KRBTEST.COM (short username kudu)
I20260925 04:54:32.240082 7991 webserver.cc:460] Webserver started at http://127.6.29.190:39715/ using document root <none> and password file <none>
I20260925 04:54:32.240828 7991 fs_manager.cc:359] Metadata directory not provided
I20260925 04:54:32.240945 7991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:54:32.241285 7991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:54:32.244359 7991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "3473cd541f7042a7903a56c2aa871845"
format_stamp: "Formatted at 2026-09-25 04:54:32 on dist-test-slave-xzjr"
server_key: "5c30c85a601d708a243c9027ad02637f"
server_key_iv: "d04855c738e7f3b207a31094e1a8052d"
server_key_version: "encryptionkey@0"
I20260925 04:54:32.245222 7991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "3473cd541f7042a7903a56c2aa871845"
format_stamp: "Formatted at 2026-09-25 04:54:32 on dist-test-slave-xzjr"
server_key: "5c30c85a601d708a243c9027ad02637f"
server_key_iv: "d04855c738e7f3b207a31094e1a8052d"
server_key_version: "encryptionkey@0"
I20260925 04:54:32.250527 7991 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 04:54:32.254005 8006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:32.255475 7991 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:54:32.255690 7991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "3473cd541f7042a7903a56c2aa871845"
format_stamp: "Formatted at 2026-09-25 04:54:32 on dist-test-slave-xzjr"
server_key: "5c30c85a601d708a243c9027ad02637f"
server_key_iv: "d04855c738e7f3b207a31094e1a8052d"
server_key_version: "encryptionkey@0"
I20260925 04:54:32.255971 7991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:54:32.291607 7991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:54:32.296370 7991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:54:32.296651 7991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:54:32.318323 7991 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:34435
I20260925 04:54:32.318344 8067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:34435 every 8 connection(s)
I20260925 04:54:32.319793 7991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:54:32.323717 8068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:54:32.330106 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 7991
I20260925 04:54:32.330370 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 04:54:32.331161 6262 external_mini_cluster.cc:1428] Setting key 761ae2704a375aa00e16ba0d87284955
I20260925 04:54:32.335219 8068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3473cd541f7042a7903a56c2aa871845: Bootstrap starting.
I20260925 04:54:32.339143 8068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3473cd541f7042a7903a56c2aa871845: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:32.340329 8068 log.cc:824] T 00000000000000000000000000000000 P 3473cd541f7042a7903a56c2aa871845: Log is configured to *not* fsync() on all Append() calls
I20260925 04:54:32.343415 8068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3473cd541f7042a7903a56c2aa871845: No bootstrap required, opened a new log
Sep 25 04:54:32 dist-test-slave-xzjr krb5kdc[7978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:54:32.351830 8068 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3473cd541f7042a7903a56c2aa871845 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3473cd541f7042a7903a56c2aa871845" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 34435 } }
I20260925 04:54:32.352246 8068 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3473cd541f7042a7903a56c2aa871845 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:32.352370 8068 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3473cd541f7042a7903a56c2aa871845 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3473cd541f7042a7903a56c2aa871845, State: Initialized, Role: FOLLOWER
I20260925 04:54:32.353098 8068 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3473cd541f7042a7903a56c2aa871845 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3473cd541f7042a7903a56c2aa871845" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 34435 } }
I20260925 04:54:32.353394 8068 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3473cd541f7042a7903a56c2aa871845 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:54:32.353562 8068 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3473cd541f7042a7903a56c2aa871845 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:54:32.353750 8068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3473cd541f7042a7903a56c2aa871845 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:32.354645 8071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:32.333425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42584 (local address 127.6.29.190:34435)
0925 04:54:32.334237 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:32.334250 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:32.334293 (+ 43us) server_negotiation.cc:407] Connection header received
0925 04:54:32.336132 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:32.336158 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:32.336574 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:32.336946 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:32.338697 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:32.340177 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:32.341747 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:32.342396 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:32.342530 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:32.347062 (+ 4532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:32.347073 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:32.347090 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:32.347133 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:32.350283 (+ 3150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:32.350987 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:32.350997 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:32.351003 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:32.351161 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:32.351547 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:32.351554 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:32.351557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:32.352225 (+ 668us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:32.352551 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:32.352980 (+ 429us) server_negotiation.cc:299] Negotiation successful
0925 04:54:32.353447 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":389,"threads_started":1}
I20260925 04:54:32.356534 8068 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3473cd541f7042a7903a56c2aa871845 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3473cd541f7042a7903a56c2aa871845" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 34435 } }
I20260925 04:54:32.357146 8068 leader_election.cc:302] T 00000000000000000000000000000000 P 3473cd541f7042a7903a56c2aa871845 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3473cd541f7042a7903a56c2aa871845; no voters:
I20260925 04:54:32.357824 8068 leader_election.cc:288] T 00000000000000000000000000000000 P 3473cd541f7042a7903a56c2aa871845 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:54:32.357939 8073 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3473cd541f7042a7903a56c2aa871845 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:54:32.358685 8073 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3473cd541f7042a7903a56c2aa871845 [term 1 LEADER]: Becoming Leader. State: Replica: 3473cd541f7042a7903a56c2aa871845, State: Running, Role: LEADER
I20260925 04:54:32.359300 8073 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3473cd541f7042a7903a56c2aa871845 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3473cd541f7042a7903a56c2aa871845" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 34435 } }
I20260925 04:54:32.360049 8068 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3473cd541f7042a7903a56c2aa871845 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:54:32.362056 8075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3473cd541f7042a7903a56c2aa871845 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3473cd541f7042a7903a56c2aa871845. Latest consensus state: current_term: 1 leader_uuid: "3473cd541f7042a7903a56c2aa871845" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3473cd541f7042a7903a56c2aa871845" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 34435 } } }
I20260925 04:54:32.362411 8075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3473cd541f7042a7903a56c2aa871845 [sys.catalog]: This master's current role is: LEADER
I20260925 04:54:32.363118 8074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3473cd541f7042a7903a56c2aa871845 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3473cd541f7042a7903a56c2aa871845" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3473cd541f7042a7903a56c2aa871845" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 34435 } } }
I20260925 04:54:32.363538 8074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3473cd541f7042a7903a56c2aa871845 [sys.catalog]: This master's current role is: LEADER
I20260925 04:54:32.368906 8078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:54:32.373863 8078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:54:32.382143 8078 catalog_manager.cc:1347] Generated new cluster ID: b7acdae9254a46cbb9e767a1e6412982
I20260925 04:54:32.382272 8078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:54:32.396354 8078 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:54:32.397806 8078 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:54:32.406217 8078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3473cd541f7042a7903a56c2aa871845: Generated new TSK 0
I20260925 04:54:32.407286 8078 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.29.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab.
Entry for principal kudu/127.6.29.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab.
I20260925 04:54:32.455725 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.129:0
--local_ip_for_outbound_sockets=127.6.29.129
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:34435
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:34565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:54:32.779666 8094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:54:32.780119 8094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:54:32.780294 8094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:54:32.787992 8094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:54:32.788162 8094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:54:32.788318 8094 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 04:54:32.788410 8094 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 04:54:32.788551 8094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.129
I20260925 04:54:32.798534 8094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:34565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.29.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.29.190:34435
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.129
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:54:32.799902 8094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:54:32.801234 8094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:54:32.814826 8099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:32.814869 8100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:32.814785 8102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:54:32.815711 8094 server_base.cc:1034] running on GCE node
I20260925 04:54:32.816365 8094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:54:32.817370 8094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:54:32.818604 8094 hybrid_clock.cc:648] HybridClock initialized: now 1790312072818565 us; error 48 us; skew 500 ppm
I20260925 04:54:32.818953 8094 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:54:32 dist-test-slave-xzjr krb5kdc[7978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312072, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:54:32.822286 8094 init.cc:377] Logged in from keytab as kudu/127.6.29.129@KRBTEST.COM (short username kudu)
I20260925 04:54:32.824019 8094 webserver.cc:460] Webserver started at http://127.6.29.129:35835/ using document root <none> and password file <none>
I20260925 04:54:32.824733 8094 fs_manager.cc:359] Metadata directory not provided
I20260925 04:54:32.824873 8094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:54:32.825201 8094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:54:32.828213 8094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-0/data/instance:
uuid: "0cb0df97fd14448d878f46990713c052"
format_stamp: "Formatted at 2026-09-25 04:54:32 on dist-test-slave-xzjr"
server_key: "d67c3b05b2bc87caf33e19beaa211fe1"
server_key_iv: "ba90488403c14a5db757ef32e7798bc0"
server_key_version: "encryptionkey@0"
I20260925 04:54:32.828971 8094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance:
uuid: "0cb0df97fd14448d878f46990713c052"
format_stamp: "Formatted at 2026-09-25 04:54:32 on dist-test-slave-xzjr"
server_key: "d67c3b05b2bc87caf33e19beaa211fe1"
server_key_iv: "ba90488403c14a5db757ef32e7798bc0"
server_key_version: "encryptionkey@0"
I20260925 04:54:32.834331 8094 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260925 04:54:32.837837 8109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:32.839128 8094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:54:32.839293 8094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-0/wal
uuid: "0cb0df97fd14448d878f46990713c052"
format_stamp: "Formatted at 2026-09-25 04:54:32 on dist-test-slave-xzjr"
server_key: "d67c3b05b2bc87caf33e19beaa211fe1"
server_key_iv: "ba90488403c14a5db757ef32e7798bc0"
server_key_version: "encryptionkey@0"
I20260925 04:54:32.839494 8094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:54:32.864264 8094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:54:32.868853 8094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:54:32.869185 8094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:54:32.871949 8094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:54:32.872099 8094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:32.872231 8094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:54:32.872354 8094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 04:54:32 dist-test-slave-xzjr krb5kdc[7978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312072, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:54:32.898969 8118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:32.872480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:50751 (local address 127.6.29.190:34435)
0925 04:54:32.873235 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:32.873242 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:32.874367 (+ 1125us) server_negotiation.cc:407] Connection header received
0925 04:54:32.877526 (+ 3159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:32.877534 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:32.877631 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:32.877734 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:32.880905 (+ 3171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:32.881887 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:32.885288 (+ 3401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:32.885860 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:32.886023 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:32.891165 (+ 5142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:32.891174 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:32.891182 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:32.891231 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:32.894837 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:32.895693 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:32.895702 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:32.895708 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:32.895902 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:32.896466 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:32.896471 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:32.896476 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:32.896890 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:32.897043 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:32.898071 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0925 04:54:32.898411 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":496,"threads_started":1}
I20260925 04:54:32.899564 8119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:32.873268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:34435 (local address 127.6.29.129:50751)
0925 04:54:32.874177 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:32.874233 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:32.877111 (+ 2878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:32.878407 (+ 1296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:32.878424 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:32.879500 (+ 1076us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:54:32.880677 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:32.880710 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:32.882067 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:32.882078 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:32.885100 (+ 3022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:32.885115 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:32.886187 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:32.886199 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:32.886584 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:32.887415 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:54:32.887461 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:54:32.890917 (+ 3456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:54:32.894989 (+ 4072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:32.895000 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:32.895023 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:32.895547 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:32.896033 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:32.896040 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:32.896048 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:32.896340 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:32.897022 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:54:32.897034 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:54:32.897558 (+ 524us) client_negotiation.cc:769] Sending connection context
0925 04:54:32.897937 (+ 379us) client_negotiation.cc:240] Negotiation successful
0925 04:54:32.898364 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":348,"threads_started":1}
I20260925 04:54:32.919534 8118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:32.906156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:48493 (local address 127.6.29.190:34435)
0925 04:54:32.906341 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:32.906349 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:32.906980 (+ 631us) server_negotiation.cc:407] Connection header received
0925 04:54:32.907171 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:32.907176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:32.907250 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:32.907359 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:32.908602 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:32.909475 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:32.910767 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:32.911161 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:32.911307 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:32.913657 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:32.913664 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:32.913668 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:32.913707 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:32.916664 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:32.917587 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:32.917594 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:32.917597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:32.917730 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:32.918082 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:32.918087 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:32.918091 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:32.918366 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:32.918509 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:32.918944 (+ 435us) server_negotiation.cc:299] Negotiation successful
0925 04:54:32.919173 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 04:54:32.919628 8194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:32.906223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:34435 (local address 127.6.29.129:48493)
0925 04:54:32.906875 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:32.906894 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:32.907050 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:32.907377 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:32.907381 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:32.907923 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:54:32.908391 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:32.908414 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:32.909601 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:32.909609 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:32.910619 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:32.910629 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:32.911341 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:32.911347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:32.911504 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:32.912212 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:54:32.912228 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:54:32.913410 (+ 1182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:54:32.916812 (+ 3402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:32.916818 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:32.916822 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:32.917452 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:32.917846 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:32.917851 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:32.917855 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:32.917977 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:32.918488 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:54:32.918492 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:54:32.918683 (+ 191us) client_negotiation.cc:769] Sending connection context
0925 04:54:32.918846 (+ 163us) client_negotiation.cc:240] Negotiation successful
0925 04:54:32.919173 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":374,"threads_started":1}
I20260925 04:54:32.927037 8094 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.129:45705
I20260925 04:54:32.927073 8229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.129:45705 every 8 connection(s)
I20260925 04:54:32.928340 8094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
I20260925 04:54:32.930229 8230 heartbeater.cc:347] Connected to a master server at 127.6.29.190:34435
I20260925 04:54:32.930418 8230 heartbeater.cc:464] Registering TS with master...
I20260925 04:54:32.930992 8230 heartbeater.cc:511] Master 127.6.29.190:34435 requested a full tablet report, sending...
I20260925 04:54:32.932796 8023 ts_manager.cc:194] Registered new tserver with Master: 0cb0df97fd14448d878f46990713c052 (127.6.29.129:45705)
I20260925 04:54:32.934183 8023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.129@KRBTEST.COM'} at 127.6.29.129:50751
I20260925 04:54:32.936293 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 8094
I20260925 04:54:32.936517 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance
I20260925 04:54:32.937096 6262 external_mini_cluster.cc:1428] Setting key fc56112f9896ade0d9143394800b35cb
WARNING: no policy specified for kudu/127.6.29.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab.
Entry for principal kudu/127.6.29.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab.
I20260925 04:54:32.979449 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.130:0
--local_ip_for_outbound_sockets=127.6.29.130
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:34435
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:34565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:54:33.309773 8234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:54:33.310067 8234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:54:33.310238 8234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:54:33.317395 8234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:54:33.317551 8234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:54:33.317678 8234 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 04:54:33.317747 8234 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 04:54:33.317847 8234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.130
I20260925 04:54:33.333235 8234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:34565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.29.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.29.190:34435
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.130
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:54:33.334851 8234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:54:33.336555 8234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:54:33.350955 8240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:33.350984 8242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:33.351027 8239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:54:33.357304 8234 server_base.cc:1034] running on GCE node
I20260925 04:54:33.358054 8234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:54:33.359099 8234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:54:33.360312 8234 hybrid_clock.cc:648] HybridClock initialized: now 1790312073360268 us; error 37 us; skew 500 ppm
I20260925 04:54:33.360692 8234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:54:33 dist-test-slave-xzjr krb5kdc[7978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312073, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:54:33.364169 8234 init.cc:377] Logged in from keytab as kudu/127.6.29.130@KRBTEST.COM (short username kudu)
I20260925 04:54:33.365764 8234 webserver.cc:460] Webserver started at http://127.6.29.130:44857/ using document root <none> and password file <none>
I20260925 04:54:33.366483 8234 fs_manager.cc:359] Metadata directory not provided
I20260925 04:54:33.366614 8234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:54:33.366948 8234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:54:33.370069 8234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-1/data/instance:
uuid: "58e38d2ed0094a75a7b48af8719ba90e"
format_stamp: "Formatted at 2026-09-25 04:54:33 on dist-test-slave-xzjr"
server_key: "c75f372b5a0f45fb7fa0411f71ba6931"
server_key_iv: "e92ecc9405e7a0903017589b7619715f"
server_key_version: "encryptionkey@0"
I20260925 04:54:33.370870 8234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance:
uuid: "58e38d2ed0094a75a7b48af8719ba90e"
format_stamp: "Formatted at 2026-09-25 04:54:33 on dist-test-slave-xzjr"
server_key: "c75f372b5a0f45fb7fa0411f71ba6931"
server_key_iv: "e92ecc9405e7a0903017589b7619715f"
server_key_version: "encryptionkey@0"
I20260925 04:54:33.376194 8234 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 04:54:33.379709 8249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:33.381263 8234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:54:33.381443 8234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-1/wal
uuid: "58e38d2ed0094a75a7b48af8719ba90e"
format_stamp: "Formatted at 2026-09-25 04:54:33 on dist-test-slave-xzjr"
server_key: "c75f372b5a0f45fb7fa0411f71ba6931"
server_key_iv: "e92ecc9405e7a0903017589b7619715f"
server_key_version: "encryptionkey@0"
I20260925 04:54:33.381632 8234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:54:33.398784 8234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:54:33.404137 8234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:54:33.404445 8234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:54:33.407192 8234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:54:33.407315 8234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:33.407444 8234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:54:33.407518 8234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 04:54:33 dist-test-slave-xzjr krb5kdc[7978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312073, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:54:33.433475 8118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:33.407932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:33019 (local address 127.6.29.190:34435)
0925 04:54:33.408105 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:33.408113 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:33.410088 (+ 1975us) server_negotiation.cc:407] Connection header received
0925 04:54:33.412682 (+ 2594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:33.412691 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:33.412805 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:33.412898 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:33.415294 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:33.416490 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:33.419100 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:33.419809 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:33.419972 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:33.425648 (+ 5676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:33.425660 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:33.425663 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:33.425697 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:33.429115 (+ 3418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:33.430404 (+ 1289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:33.430411 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:33.430417 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:33.430579 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:33.431209 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:33.431215 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:33.431218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:33.431616 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:33.431813 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:33.432882 (+ 1069us) server_negotiation.cc:299] Negotiation successful
0925 04:54:33.433105 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 04:54:33.435107 8258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:33.408999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:34435 (local address 127.6.29.130:33019)
0925 04:54:33.409921 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:33.409966 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:33.412321 (+ 2355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:33.413327 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:33.413335 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:33.414114 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:54:33.415094 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:33.415126 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:33.416644 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:33.416652 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:33.418940 (+ 2288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:33.418952 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:33.420068 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:33.420079 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:33.420498 (+ 419us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:33.421452 (+ 954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:54:33.421501 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:54:33.425362 (+ 3861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:54:33.429291 (+ 3929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:33.429305 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:33.429329 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:33.430242 (+ 913us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:33.430703 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:33.430708 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:33.430712 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:33.431022 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:33.431895 (+ 873us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:54:33.431904 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:54:33.432485 (+ 581us) client_negotiation.cc:769] Sending connection context
0925 04:54:33.432810 (+ 325us) client_negotiation.cc:240] Negotiation successful
0925 04:54:33.433403 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":391,"threads_started":1}
I20260925 04:54:33.454697 8234 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.130:40465
I20260925 04:54:33.454705 8368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.130:40465 every 8 connection(s)
I20260925 04:54:33.456087 8234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
I20260925 04:54:33.457716 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 8234
I20260925 04:54:33.457921 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance
I20260925 04:54:33.458444 6262 external_mini_cluster.cc:1428] Setting key ed751d0170256fd1558a6b355b90431b
I20260925 04:54:33.458595 8118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:33.442553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:55531 (local address 127.6.29.190:34435)
0925 04:54:33.442761 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:33.442767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:33.443588 (+ 821us) server_negotiation.cc:407] Connection header received
0925 04:54:33.443911 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:33.443919 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:33.444015 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:33.444124 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:33.445751 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:33.446765 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:33.449551 (+ 2786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:33.450137 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:33.450324 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:33.452869 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:33.452873 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:33.452876 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:33.452907 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:33.455714 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:33.456477 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:33.456487 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:33.456489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:33.456616 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:33.457122 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:33.457128 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:33.457132 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:33.457561 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:33.457682 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:33.458164 (+ 482us) server_negotiation.cc:299] Negotiation successful
0925 04:54:33.458332 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 04:54:33.458822 8348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:33.442609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:34435 (local address 127.6.29.130:55531)
0925 04:54:33.443458 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:33.443485 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:33.443727 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:33.444234 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:33.444242 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:33.445070 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:54:33.445553 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:33.445570 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:33.446944 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:33.446978 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:33.449344 (+ 2366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:33.449367 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:33.450383 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:33.450390 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:33.450573 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:33.451374 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:54:33.451391 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:54:33.452664 (+ 1273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:54:33.455877 (+ 3213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:33.455884 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:33.455888 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:33.456353 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:33.456741 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:33.456752 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:33.456756 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:33.456959 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:33.457694 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:54:33.457701 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:54:33.457908 (+ 207us) client_negotiation.cc:769] Sending connection context
0925 04:54:33.458063 (+ 155us) client_negotiation.cc:240] Negotiation successful
0925 04:54:33.458390 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":462,"threads_started":1}
I20260925 04:54:33.460258 8369 heartbeater.cc:347] Connected to a master server at 127.6.29.190:34435
I20260925 04:54:33.460484 8369 heartbeater.cc:464] Registering TS with master...
I20260925 04:54:33.461400 8369 heartbeater.cc:511] Master 127.6.29.190:34435 requested a full tablet report, sending...
I20260925 04:54:33.462685 8022 ts_manager.cc:194] Registered new tserver with Master: 58e38d2ed0094a75a7b48af8719ba90e (127.6.29.130:40465)
I20260925 04:54:33.463711 8022 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.130@KRBTEST.COM'} at 127.6.29.130:33019
WARNING: no policy specified for kudu/127.6.29.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab.
Entry for principal kudu/127.6.29.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab.
I20260925 04:54:33.504335 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.131:0
--local_ip_for_outbound_sockets=127.6.29.131
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:34435
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:34565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:54:33.823422 8373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:54:33.823709 8373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:54:33.823909 8373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:54:33.831175 8373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:54:33.831280 8373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:54:33.831470 8373 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 04:54:33.831552 8373 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 04:54:33.831683 8373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.131
I20260925 04:54:33.841511 8373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:34565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.29.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.29.190:34435
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.131
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:54:33.842937 8373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:54:33.844396 8373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:54:33.857625 8378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:33.857671 8379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:33.857712 8381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:54:33.858585 8373 server_base.cc:1034] running on GCE node
I20260925 04:54:33.859176 8373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:54:33.860345 8373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:54:33.861544 8373 hybrid_clock.cc:648] HybridClock initialized: now 1790312073861492 us; error 45 us; skew 500 ppm
I20260925 04:54:33.861824 8373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:54:33 dist-test-slave-xzjr krb5kdc[7978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312073, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:54:33.865079 8373 init.cc:377] Logged in from keytab as kudu/127.6.29.131@KRBTEST.COM (short username kudu)
I20260925 04:54:33.866604 8373 webserver.cc:460] Webserver started at http://127.6.29.131:35831/ using document root <none> and password file <none>
I20260925 04:54:33.867317 8373 fs_manager.cc:359] Metadata directory not provided
I20260925 04:54:33.867452 8373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:54:33.867884 8373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:54:33.870994 8373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-2/data/instance:
uuid: "3ff76b5d25ad47a5897e7bce347ea4bc"
format_stamp: "Formatted at 2026-09-25 04:54:33 on dist-test-slave-xzjr"
server_key: "1f7f71c1a354ca450c35caac805cb469"
server_key_iv: "4e446029d891be1165c4263bcdb5709f"
server_key_version: "encryptionkey@0"
I20260925 04:54:33.871812 8373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance:
uuid: "3ff76b5d25ad47a5897e7bce347ea4bc"
format_stamp: "Formatted at 2026-09-25 04:54:33 on dist-test-slave-xzjr"
server_key: "1f7f71c1a354ca450c35caac805cb469"
server_key_iv: "4e446029d891be1165c4263bcdb5709f"
server_key_version: "encryptionkey@0"
I20260925 04:54:33.877204 8373 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 04:54:33.880749 8388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:33.882105 8373 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 04:54:33.882268 8373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-2/wal
uuid: "3ff76b5d25ad47a5897e7bce347ea4bc"
format_stamp: "Formatted at 2026-09-25 04:54:33 on dist-test-slave-xzjr"
server_key: "1f7f71c1a354ca450c35caac805cb469"
server_key_iv: "4e446029d891be1165c4263bcdb5709f"
server_key_version: "encryptionkey@0"
I20260925 04:54:33.882486 8373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:54:33.901332 8373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:54:33.906004 8373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:54:33.906324 8373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:54:33.909204 8373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:54:33.909332 8373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:33.909451 8373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:54:33.909574 8373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 04:54:33 dist-test-slave-xzjr krb5kdc[7978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312073, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:54:33.934195 8118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:33.910118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:46741 (local address 127.6.29.190:34435)
0925 04:54:33.910295 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:33.910299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:33.911670 (+ 1371us) server_negotiation.cc:407] Connection header received
0925 04:54:33.913749 (+ 2079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:33.913755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:33.913854 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:33.913968 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:33.916319 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:33.917656 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:33.920118 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:33.920780 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:33.920939 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:33.926086 (+ 5147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:33.926091 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:33.926094 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:33.926128 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:33.929588 (+ 3460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:33.930813 (+ 1225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:33.930821 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:33.930826 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:33.930994 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:33.931614 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:33.931620 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:33.931623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:33.932046 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:33.932190 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:33.933305 (+ 1115us) server_negotiation.cc:299] Negotiation successful
0925 04:54:33.933658 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 04:54:33.934942 8397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:33.910657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:34435 (local address 127.6.29.131:46741)
0925 04:54:33.911496 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:33.911540 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:33.913464 (+ 1924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:33.914252 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:33.914268 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:33.915104 (+ 836us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:54:33.916066 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:33.916101 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:33.917834 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:33.917838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:33.919954 (+ 2116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:33.919966 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:33.921274 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:33.921281 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:33.921501 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:33.922409 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:54:33.922450 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:54:33.925821 (+ 3371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:54:33.929770 (+ 3949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:33.929785 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:33.929808 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:33.930636 (+ 828us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:33.931140 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:33.931148 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:33.931155 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:33.931461 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:33.932265 (+ 804us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:54:33.932273 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:54:33.932743 (+ 470us) client_negotiation.cc:769] Sending connection context
0925 04:54:33.933059 (+ 316us) client_negotiation.cc:240] Negotiation successful
0925 04:54:33.933605 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":377,"threads_started":1}
I20260925 04:54:33.936892 8230 heartbeater.cc:503] Master 127.6.29.190:34435 was elected leader, sending a full tablet report...
I20260925 04:54:33.959064 8118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:33.942526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:36585 (local address 127.6.29.190:34435)
0925 04:54:33.942754 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:33.942760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:33.943613 (+ 853us) server_negotiation.cc:407] Connection header received
0925 04:54:33.943908 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:33.943912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:33.944013 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:33.944113 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:33.945627 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:33.946439 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:33.949001 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:33.949558 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:33.949712 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:33.952566 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:33.952576 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:33.952584 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:33.952634 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:33.955978 (+ 3344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:33.956960 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:33.956965 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:33.956969 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:33.957110 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:33.957489 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:33.957494 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:33.957498 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:33.957800 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:33.957954 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:33.958398 (+ 444us) server_negotiation.cc:299] Negotiation successful
0925 04:54:33.958630 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 04:54:33.959157 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:33.942510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:34435 (local address 127.6.29.131:36585)
0925 04:54:33.943481 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:33.943507 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:33.943691 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:33.944170 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:33.944177 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:33.944945 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:54:33.945449 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:33.945465 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:33.946585 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:33.946593 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:33.948744 (+ 2151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:33.948764 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:33.949776 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:33.949787 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:33.950043 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:33.950854 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:54:33.950876 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:54:33.952352 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:54:33.956151 (+ 3799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:33.956158 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:33.956167 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:33.956819 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:33.957223 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:33.957228 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:33.957231 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:33.957366 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:33.957919 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:54:33.957924 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:54:33.958108 (+ 184us) client_negotiation.cc:769] Sending connection context
0925 04:54:33.958279 (+ 171us) client_negotiation.cc:240] Negotiation successful
0925 04:54:33.958630 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"thread_start_us":604,"threads_started":1}
I20260925 04:54:33.971038 8373 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.131:37063
I20260925 04:54:33.971078 8507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.131:37063 every 8 connection(s)
I20260925 04:54:33.972422 8373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
I20260925 04:54:33.974589 8508 heartbeater.cc:347] Connected to a master server at 127.6.29.190:34435
I20260925 04:54:33.974879 8508 heartbeater.cc:464] Registering TS with master...
I20260925 04:54:33.975549 8508 heartbeater.cc:511] Master 127.6.29.190:34435 requested a full tablet report, sending...
I20260925 04:54:33.976933 8022 ts_manager.cc:194] Registered new tserver with Master: 3ff76b5d25ad47a5897e7bce347ea4bc (127.6.29.131:37063)
I20260925 04:54:33.977769 8022 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.131@KRBTEST.COM'} at 127.6.29.131:46741
I20260925 04:54:33.982085 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 8373
I20260925 04:54:33.982236 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance
I20260925 04:54:33.982594 6262 external_mini_cluster.cc:1428] Setting key 35555beb897ee06f261fe086aa769e43
I20260925 04:54:33.985299 6262 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 04:54:34 dist-test-slave-xzjr krb5kdc[7978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 04:54:34 dist-test-slave-xzjr krb5kdc[7978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:54:34.033553 8118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:34.019872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42592 (local address 127.6.29.190:34435)
0925 04:54:34.020100 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:34.020107 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:34.020296 (+ 189us) server_negotiation.cc:407] Connection header received
0925 04:54:34.020511 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:34.020517 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:34.020587 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:34.020695 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:34.021994 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.022719 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.024107 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.024537 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.024666 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:34.028566 (+ 3900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:34.028572 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:34.028575 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:34.028607 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:34.030958 (+ 2351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:34.031607 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:34.031612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:34.031618 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:34.031737 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:34.032123 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:34.032128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:34.032131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:34.032432 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:34.032680 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:34.032980 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 04:54:34.033208 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 04:54:34.037533 8022 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:54:34.044320 8022 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:54:34.078768 8522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:34.055866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:45705 (local address 127.0.0.1:59142)
0925 04:54:34.067139 (+ 11273us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:34.067203 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:34.067436 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:34.071887 (+ 4451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:34.071895 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:34.071935 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:54:34.072333 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:34.072345 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.074075 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.074083 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:34.075572 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:34.075584 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.077741 (+ 2157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.077747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:34.078011 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:34.078074 (+ 63us) client_negotiation.cc:769] Sending connection context
0925 04:54:34.078219 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 04:54:34.078411 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10882,"thread_start_us":11805,"threads_started":1}
I20260925 04:54:34.078804 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:34.056024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59142 (local address 127.6.29.129:45705)
0925 04:54:34.071301 (+ 15277us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:34.071308 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:34.071334 (+ 26us) server_negotiation.cc:407] Connection header received
0925 04:54:34.071423 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:34.071429 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:34.071630 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:34.071861 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:54:34.072512 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.073951 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.075722 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.077198 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.077343 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:34.077586 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:34.078273 (+ 687us) server_negotiation.cc:299] Negotiation successful
0925 04:54:34.078461 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15139,"thread_start_us":16007,"threads_started":1}
I20260925 04:54:34.083341 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:34.056746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39966 (local address 127.6.29.131:37063)
0925 04:54:34.075739 (+ 18993us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:34.075752 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:34.075808 (+ 56us) server_negotiation.cc:407] Connection header received
0925 04:54:34.075922 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:34.075933 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:34.076259 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:34.077046 (+ 787us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:54:34.077679 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.079499 (+ 1820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.081383 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.082312 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.082460 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:34.082692 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:34.082887 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 04:54:34.083036 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18770,"thread_start_us":18587,"threads_started":1}
I20260925 04:54:34.083621 8521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:34.055959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.131:37063 (local address 127.0.0.1:39966)
0925 04:54:34.072854 (+ 16895us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:34.072881 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:34.072984 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:34.077033 (+ 4049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:34.077037 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:34.077061 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:54:34.077448 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:34.077460 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.079646 (+ 2186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.079651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:34.081210 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:34.081221 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.082512 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.082518 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:34.082688 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:34.082727 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 04:54:34.082837 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 04:54:34.082959 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16778,"thread_start_us":16615,"threads_started":1}
I20260925 04:54:34.083237 8160 tablet_service.cc:1467] Processing CreateTablet for tablet 7ae3064d66a74841887816fb83f6e557 (DEFAULT_TABLE table=test-table [id=080df66c52c74ed4b083745f65f39623]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:54:34.085492 8524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:34.056425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:40465 (local address 127.0.0.1:57260)
0925 04:54:34.077849 (+ 21424us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:34.077864 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:34.078006 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:34.078883 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:34.078888 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:34.078920 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:54:34.079265 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:34.079276 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.081038 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.081043 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:34.083034 (+ 1991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:34.083048 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.084755 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.084761 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:34.084932 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:34.085018 (+ 86us) client_negotiation.cc:769] Sending connection context
0925 04:54:34.085095 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 04:54:34.085178 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21300,"thread_start_us":21143,"threads_started":1}
I20260925 04:54:34.085615 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:34.057266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57260 (local address 127.6.29.130:40465)
0925 04:54:34.076855 (+ 19589us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:34.076866 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:34.077972 (+ 1106us) server_negotiation.cc:407] Connection header received
0925 04:54:34.078169 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:34.078175 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:34.078431 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:34.078626 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:54:34.079466 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.080853 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.083212 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.084551 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.084684 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:34.084923 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:34.085177 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 04:54:34.085341 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19443,"thread_start_us":19319,"threads_started":1}
I20260925 04:54:34.086905 8160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ae3064d66a74841887816fb83f6e557. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:54:34.087452 8438 tablet_service.cc:1467] Processing CreateTablet for tablet 7ae3064d66a74841887816fb83f6e557 (DEFAULT_TABLE table=test-table [id=080df66c52c74ed4b083745f65f39623]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:54:34.088838 8438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ae3064d66a74841887816fb83f6e557. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:54:34.088953 8299 tablet_service.cc:1467] Processing CreateTablet for tablet 7ae3064d66a74841887816fb83f6e557 (DEFAULT_TABLE table=test-table [id=080df66c52c74ed4b083745f65f39623]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:54:34.090220 8299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ae3064d66a74841887816fb83f6e557. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:54:34.099891 8527 tablet_bootstrap.cc:492] T 7ae3064d66a74841887816fb83f6e557 P 3ff76b5d25ad47a5897e7bce347ea4bc: Bootstrap starting.
I20260925 04:54:34.101943 8528 tablet_bootstrap.cc:492] T 7ae3064d66a74841887816fb83f6e557 P 0cb0df97fd14448d878f46990713c052: Bootstrap starting.
I20260925 04:54:34.104480 8529 tablet_bootstrap.cc:492] T 7ae3064d66a74841887816fb83f6e557 P 58e38d2ed0094a75a7b48af8719ba90e: Bootstrap starting.
I20260925 04:54:34.104794 8527 tablet_bootstrap.cc:654] T 7ae3064d66a74841887816fb83f6e557 P 3ff76b5d25ad47a5897e7bce347ea4bc: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:34.105731 8528 tablet_bootstrap.cc:654] T 7ae3064d66a74841887816fb83f6e557 P 0cb0df97fd14448d878f46990713c052: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:34.106019 8527 log.cc:824] T 7ae3064d66a74841887816fb83f6e557 P 3ff76b5d25ad47a5897e7bce347ea4bc: Log is configured to *not* fsync() on all Append() calls
I20260925 04:54:34.106983 8528 log.cc:824] T 7ae3064d66a74841887816fb83f6e557 P 0cb0df97fd14448d878f46990713c052: Log is configured to *not* fsync() on all Append() calls
I20260925 04:54:34.107931 8529 tablet_bootstrap.cc:654] T 7ae3064d66a74841887816fb83f6e557 P 58e38d2ed0094a75a7b48af8719ba90e: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:34.108954 8527 tablet_bootstrap.cc:492] T 7ae3064d66a74841887816fb83f6e557 P 3ff76b5d25ad47a5897e7bce347ea4bc: No bootstrap required, opened a new log
I20260925 04:54:34.109117 8529 log.cc:824] T 7ae3064d66a74841887816fb83f6e557 P 58e38d2ed0094a75a7b48af8719ba90e: Log is configured to *not* fsync() on all Append() calls
I20260925 04:54:34.109391 8527 ts_tablet_manager.cc:1397] T 7ae3064d66a74841887816fb83f6e557 P 3ff76b5d25ad47a5897e7bce347ea4bc: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260925 04:54:34.109939 8528 tablet_bootstrap.cc:492] T 7ae3064d66a74841887816fb83f6e557 P 0cb0df97fd14448d878f46990713c052: No bootstrap required, opened a new log
I20260925 04:54:34.110231 8528 ts_tablet_manager.cc:1397] T 7ae3064d66a74841887816fb83f6e557 P 0cb0df97fd14448d878f46990713c052: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260925 04:54:34.112699 8529 tablet_bootstrap.cc:492] T 7ae3064d66a74841887816fb83f6e557 P 58e38d2ed0094a75a7b48af8719ba90e: No bootstrap required, opened a new log
I20260925 04:54:34.112962 8529 ts_tablet_manager.cc:1397] T 7ae3064d66a74841887816fb83f6e557 P 58e38d2ed0094a75a7b48af8719ba90e: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260925 04:54:34.117188 8527 raft_consensus.cc:348] T 7ae3064d66a74841887816fb83f6e557 P 3ff76b5d25ad47a5897e7bce347ea4bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } }
I20260925 04:54:34.117597 8527 raft_consensus.cc:374] T 7ae3064d66a74841887816fb83f6e557 P 3ff76b5d25ad47a5897e7bce347ea4bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:34.117697 8527 raft_consensus.cc:729] T 7ae3064d66a74841887816fb83f6e557 P 3ff76b5d25ad47a5897e7bce347ea4bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ff76b5d25ad47a5897e7bce347ea4bc, State: Initialized, Role: FOLLOWER
I20260925 04:54:34.118340 8527 consensus_queue.cc:260] T 7ae3064d66a74841887816fb83f6e557 P 3ff76b5d25ad47a5897e7bce347ea4bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } }
I20260925 04:54:34.119421 8508 heartbeater.cc:503] Master 127.6.29.190:34435 was elected leader, sending a full tablet report...
I20260925 04:54:34.120709 8527 ts_tablet_manager.cc:1428] T 7ae3064d66a74841887816fb83f6e557 P 3ff76b5d25ad47a5897e7bce347ea4bc: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260925 04:54:34.120899 8528 raft_consensus.cc:348] T 7ae3064d66a74841887816fb83f6e557 P 0cb0df97fd14448d878f46990713c052 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } }
I20260925 04:54:34.121315 8528 raft_consensus.cc:374] T 7ae3064d66a74841887816fb83f6e557 P 0cb0df97fd14448d878f46990713c052 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:34.121397 8528 raft_consensus.cc:729] T 7ae3064d66a74841887816fb83f6e557 P 0cb0df97fd14448d878f46990713c052 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb0df97fd14448d878f46990713c052, State: Initialized, Role: FOLLOWER
I20260925 04:54:34.121973 8528 consensus_queue.cc:260] T 7ae3064d66a74841887816fb83f6e557 P 0cb0df97fd14448d878f46990713c052 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } }
I20260925 04:54:34.123980 8529 raft_consensus.cc:348] T 7ae3064d66a74841887816fb83f6e557 P 58e38d2ed0094a75a7b48af8719ba90e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } }
I20260925 04:54:34.124439 8529 raft_consensus.cc:374] T 7ae3064d66a74841887816fb83f6e557 P 58e38d2ed0094a75a7b48af8719ba90e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:34.124536 8529 raft_consensus.cc:729] T 7ae3064d66a74841887816fb83f6e557 P 58e38d2ed0094a75a7b48af8719ba90e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58e38d2ed0094a75a7b48af8719ba90e, State: Initialized, Role: FOLLOWER
I20260925 04:54:34.125218 8528 ts_tablet_manager.cc:1428] T 7ae3064d66a74841887816fb83f6e557 P 0cb0df97fd14448d878f46990713c052: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
I20260925 04:54:34.125226 8529 consensus_queue.cc:260] T 7ae3064d66a74841887816fb83f6e557 P 58e38d2ed0094a75a7b48af8719ba90e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } }
I20260925 04:54:34.127432 8369 heartbeater.cc:503] Master 127.6.29.190:34435 was elected leader, sending a full tablet report...
I20260925 04:54:34.129097 8529 ts_tablet_manager.cc:1428] T 7ae3064d66a74841887816fb83f6e557 P 58e38d2ed0094a75a7b48af8719ba90e: Time spent starting tablet: real 0.016s user 0.008s sys 0.007s
W20260925 04:54:34.180634 8231 tablet.cc:2369] T 7ae3064d66a74841887816fb83f6e557 P 0cb0df97fd14448d878f46990713c052: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:54:34.185199 8534 raft_consensus.cc:482] T 7ae3064d66a74841887816fb83f6e557 P 0cb0df97fd14448d878f46990713c052 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:54:34.185454 8534 raft_consensus.cc:504] T 7ae3064d66a74841887816fb83f6e557 P 0cb0df97fd14448d878f46990713c052 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } }
I20260925 04:54:34.187120 8534 leader_election.cc:288] T 7ae3064d66a74841887816fb83f6e557 P 0cb0df97fd14448d878f46990713c052 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ff76b5d25ad47a5897e7bce347ea4bc (127.6.29.131:37063), 58e38d2ed0094a75a7b48af8719ba90e (127.6.29.130:40465)
I20260925 04:54:34.196249 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:34.187617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:33169 (local address 127.6.29.131:37063)
0925 04:54:34.187827 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:34.187833 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:34.188575 (+ 742us) server_negotiation.cc:407] Connection header received
0925 04:54:34.188751 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:34.188756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:34.188859 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:34.188948 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:54:34.189756 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.190848 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.193886 (+ 3038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.195110 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.195281 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:34.195357 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:34.195850 (+ 493us) server_negotiation.cc:299] Negotiation successful
0925 04:54:34.195978 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 04:54:34.196524 8537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:34.187398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.131:37063 (local address 127.6.29.129:33169)
0925 04:54:34.188405 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:34.188419 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:34.188612 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:34.188995 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:34.189000 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:34.189034 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:54:34.189618 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:34.189626 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.191003 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.191009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:34.193694 (+ 2685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:34.193710 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.195390 (+ 1680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.195395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:34.195602 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:34.195704 (+ 102us) client_negotiation.cc:769] Sending connection context
0925 04:54:34.195823 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 04:54:34.196055 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":823,"mutex_wait_us":375,"spinlock_wait_cycles":42624,"thread_start_us":607,"threads_started":1}
I20260925 04:54:34.196852 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:34.187617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:38197 (local address 127.6.29.130:40465)
0925 04:54:34.187827 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:34.187833 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:34.188526 (+ 693us) server_negotiation.cc:407] Connection header received
0925 04:54:34.188739 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:34.188745 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:34.188824 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:34.188929 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:54:34.189756 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.190877 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.193878 (+ 3001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.195113 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.195286 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:34.195368 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:34.196493 (+ 1125us) server_negotiation.cc:299] Negotiation successful
0925 04:54:34.196619 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 04:54:34.197134 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:34.187516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:40465 (local address 127.6.29.129:38197)
0925 04:54:34.188394 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:34.188417 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:34.188574 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:34.188995 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:34.189000 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:34.189039 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:54:34.189543 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:34.189561 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.191005 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.191009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:34.193694 (+ 2685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:34.193710 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.195351 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.195358 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:34.195605 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:34.195699 (+ 94us) client_negotiation.cc:769] Sending connection context
0925 04:54:34.195823 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 04:54:34.196055 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"mutex_wait_us":322,"spinlock_wait_cycles":6144,"thread_start_us":490,"threads_started":1}
I20260925 04:54:34.197715 8463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ae3064d66a74841887816fb83f6e557" candidate_uuid: "0cb0df97fd14448d878f46990713c052" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" is_pre_election: true
I20260925 04:54:34.198347 8463 raft_consensus.cc:2454] T 7ae3064d66a74841887816fb83f6e557 P 3ff76b5d25ad47a5897e7bce347ea4bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cb0df97fd14448d878f46990713c052 in term 0.
I20260925 04:54:34.198236 8319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ae3064d66a74841887816fb83f6e557" candidate_uuid: "0cb0df97fd14448d878f46990713c052" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58e38d2ed0094a75a7b48af8719ba90e" is_pre_election: true
I20260925 04:54:34.198786 8319 raft_consensus.cc:2454] T 7ae3064d66a74841887816fb83f6e557 P 58e38d2ed0094a75a7b48af8719ba90e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cb0df97fd14448d878f46990713c052 in term 0.
I20260925 04:54:34.199088 8112 leader_election.cc:302] T 7ae3064d66a74841887816fb83f6e557 P 0cb0df97fd14448d878f46990713c052 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cb0df97fd14448d878f46990713c052, 3ff76b5d25ad47a5897e7bce347ea4bc; no voters:
I20260925 04:54:34.199539 8534 raft_consensus.cc:2781] T 7ae3064d66a74841887816fb83f6e557 P 0cb0df97fd14448d878f46990713c052 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:54:34.199700 8534 raft_consensus.cc:482] T 7ae3064d66a74841887816fb83f6e557 P 0cb0df97fd14448d878f46990713c052 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:54:34.199841 8534 raft_consensus.cc:3037] T 7ae3064d66a74841887816fb83f6e557 P 0cb0df97fd14448d878f46990713c052 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:34.202142 8534 raft_consensus.cc:504] T 7ae3064d66a74841887816fb83f6e557 P 0cb0df97fd14448d878f46990713c052 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } }
I20260925 04:54:34.202854 8534 leader_election.cc:288] T 7ae3064d66a74841887816fb83f6e557 P 0cb0df97fd14448d878f46990713c052 [CANDIDATE]: Term 1 election: Requested vote from peers 3ff76b5d25ad47a5897e7bce347ea4bc (127.6.29.131:37063), 58e38d2ed0094a75a7b48af8719ba90e (127.6.29.130:40465)
I20260925 04:54:34.203267 8463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ae3064d66a74841887816fb83f6e557" candidate_uuid: "0cb0df97fd14448d878f46990713c052" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc"
I20260925 04:54:34.203363 8319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ae3064d66a74841887816fb83f6e557" candidate_uuid: "0cb0df97fd14448d878f46990713c052" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58e38d2ed0094a75a7b48af8719ba90e"
I20260925 04:54:34.203469 8463 raft_consensus.cc:3037] T 7ae3064d66a74841887816fb83f6e557 P 3ff76b5d25ad47a5897e7bce347ea4bc [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:34.203601 8319 raft_consensus.cc:3037] T 7ae3064d66a74841887816fb83f6e557 P 58e38d2ed0094a75a7b48af8719ba90e [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:34.205749 8463 raft_consensus.cc:2454] T 7ae3064d66a74841887816fb83f6e557 P 3ff76b5d25ad47a5897e7bce347ea4bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cb0df97fd14448d878f46990713c052 in term 1.
I20260925 04:54:34.205865 8319 raft_consensus.cc:2454] T 7ae3064d66a74841887816fb83f6e557 P 58e38d2ed0094a75a7b48af8719ba90e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cb0df97fd14448d878f46990713c052 in term 1.
I20260925 04:54:34.206203 8112 leader_election.cc:302] T 7ae3064d66a74841887816fb83f6e557 P 0cb0df97fd14448d878f46990713c052 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cb0df97fd14448d878f46990713c052, 3ff76b5d25ad47a5897e7bce347ea4bc; no voters:
I20260925 04:54:34.206468 8534 raft_consensus.cc:2781] T 7ae3064d66a74841887816fb83f6e557 P 0cb0df97fd14448d878f46990713c052 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:54:34.206912 8534 raft_consensus.cc:686] T 7ae3064d66a74841887816fb83f6e557 P 0cb0df97fd14448d878f46990713c052 [term 1 LEADER]: Becoming Leader. State: Replica: 0cb0df97fd14448d878f46990713c052, State: Running, Role: LEADER
I20260925 04:54:34.207422 8534 consensus_queue.cc:237] T 7ae3064d66a74841887816fb83f6e557 P 0cb0df97fd14448d878f46990713c052 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } }
W20260925 04:54:34.208135 8370 tablet.cc:2369] T 7ae3064d66a74841887816fb83f6e557 P 58e38d2ed0094a75a7b48af8719ba90e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:54:34.212034 8023 catalog_manager.cc:5495] T 7ae3064d66a74841887816fb83f6e557 P 0cb0df97fd14448d878f46990713c052 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cb0df97fd14448d878f46990713c052 (127.6.29.129). New cstate: current_term: 1 leader_uuid: "0cb0df97fd14448d878f46990713c052" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } health_report { overall_health: UNKNOWN } } }
W20260925 04:54:34.224124 8509 tablet.cc:2369] T 7ae3064d66a74841887816fb83f6e557 P 3ff76b5d25ad47a5897e7bce347ea4bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 04:54:34 dist-test-slave-xzjr krb5kdc[7978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312072, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:54:34.271602 8545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:34.258182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:34435 (local address 127.0.0.1:42598)
0925 04:54:34.258798 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:34.258814 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:34.259097 (+ 283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:34.259503 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:34.259507 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:34.260331 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:54:34.260670 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:34.260681 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.261665 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.261673 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:34.262632 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:34.262642 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.263395 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.263400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:34.263560 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:34.264272 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:54:34.264288 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:54:34.266420 (+ 2132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:54:34.269477 (+ 3057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:34.269487 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:34.269493 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:34.269882 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:34.270180 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:34.270183 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:34.270185 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:34.270342 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:34.270776 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:54:34.270781 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:54:34.270925 (+ 144us) client_negotiation.cc:769] Sending connection context
0925 04:54:34.271054 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 04:54:34.271246 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":338,"threads_started":1}
I20260925 04:54:34.271608 8118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:34.258181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42598 (local address 127.6.29.190:34435)
0925 04:54:34.258360 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:34.258365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:34.259008 (+ 643us) server_negotiation.cc:407] Connection header received
0925 04:54:34.259218 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:34.259225 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:34.259294 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:34.259478 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:34.260798 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.261541 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.262770 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.263203 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.263344 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:34.266588 (+ 3244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:34.266593 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:34.266597 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:34.266626 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:34.269363 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:34.269976 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:34.269980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:34.269982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:34.270092 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:34.270426 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:34.270429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:34.270431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:34.270676 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:34.270777 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:34.271141 (+ 364us) server_negotiation.cc:299] Negotiation successful
0925 04:54:34.271359 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":65}
I20260925 04:54:34.276614 8023 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:42598:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 04:54:34.277714 8023 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:54:34.284319 8160 tablet_service.cc:1467] Processing CreateTablet for tablet f8016f4a12884d69bc89feaabd86f9fa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cae308fc942e47d6b8afd76420439a35]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 04:54:34.284654 8299 tablet_service.cc:1467] Processing CreateTablet for tablet f8016f4a12884d69bc89feaabd86f9fa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cae308fc942e47d6b8afd76420439a35]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 04:54:34.284672 8438 tablet_service.cc:1467] Processing CreateTablet for tablet f8016f4a12884d69bc89feaabd86f9fa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cae308fc942e47d6b8afd76420439a35]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 04:54:34.285398 8160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8016f4a12884d69bc89feaabd86f9fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:54:34.285478 8299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8016f4a12884d69bc89feaabd86f9fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:54:34.285508 8438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8016f4a12884d69bc89feaabd86f9fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:54:34.291112 8527 tablet_bootstrap.cc:492] T f8016f4a12884d69bc89feaabd86f9fa P 3ff76b5d25ad47a5897e7bce347ea4bc: Bootstrap starting.
I20260925 04:54:34.292428 8529 tablet_bootstrap.cc:492] T f8016f4a12884d69bc89feaabd86f9fa P 58e38d2ed0094a75a7b48af8719ba90e: Bootstrap starting.
I20260925 04:54:34.292501 8528 tablet_bootstrap.cc:492] T f8016f4a12884d69bc89feaabd86f9fa P 0cb0df97fd14448d878f46990713c052: Bootstrap starting.
I20260925 04:54:34.292804 8527 tablet_bootstrap.cc:654] T f8016f4a12884d69bc89feaabd86f9fa P 3ff76b5d25ad47a5897e7bce347ea4bc: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:34.294085 8528 tablet_bootstrap.cc:654] T f8016f4a12884d69bc89feaabd86f9fa P 0cb0df97fd14448d878f46990713c052: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:34.294162 8529 tablet_bootstrap.cc:654] T f8016f4a12884d69bc89feaabd86f9fa P 58e38d2ed0094a75a7b48af8719ba90e: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:34.295353 8527 tablet_bootstrap.cc:492] T f8016f4a12884d69bc89feaabd86f9fa P 3ff76b5d25ad47a5897e7bce347ea4bc: No bootstrap required, opened a new log
I20260925 04:54:34.295570 8527 ts_tablet_manager.cc:1397] T f8016f4a12884d69bc89feaabd86f9fa P 3ff76b5d25ad47a5897e7bce347ea4bc: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 04:54:34.296464 8529 tablet_bootstrap.cc:492] T f8016f4a12884d69bc89feaabd86f9fa P 58e38d2ed0094a75a7b48af8719ba90e: No bootstrap required, opened a new log
I20260925 04:54:34.296553 8528 tablet_bootstrap.cc:492] T f8016f4a12884d69bc89feaabd86f9fa P 0cb0df97fd14448d878f46990713c052: No bootstrap required, opened a new log
I20260925 04:54:34.296413 8527 raft_consensus.cc:348] T f8016f4a12884d69bc89feaabd86f9fa P 3ff76b5d25ad47a5897e7bce347ea4bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } }
I20260925 04:54:34.296599 8529 ts_tablet_manager.cc:1397] T f8016f4a12884d69bc89feaabd86f9fa P 58e38d2ed0094a75a7b48af8719ba90e: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260925 04:54:34.296651 8527 raft_consensus.cc:374] T f8016f4a12884d69bc89feaabd86f9fa P 3ff76b5d25ad47a5897e7bce347ea4bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:34.296684 8528 ts_tablet_manager.cc:1397] T f8016f4a12884d69bc89feaabd86f9fa P 0cb0df97fd14448d878f46990713c052: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s
I20260925 04:54:34.296751 8527 raft_consensus.cc:729] T f8016f4a12884d69bc89feaabd86f9fa P 3ff76b5d25ad47a5897e7bce347ea4bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ff76b5d25ad47a5897e7bce347ea4bc, State: Initialized, Role: FOLLOWER
I20260925 04:54:34.296988 8527 consensus_queue.cc:260] T f8016f4a12884d69bc89feaabd86f9fa P 3ff76b5d25ad47a5897e7bce347ea4bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } }
I20260925 04:54:34.297346 8529 raft_consensus.cc:348] T f8016f4a12884d69bc89feaabd86f9fa P 58e38d2ed0094a75a7b48af8719ba90e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } }
I20260925 04:54:34.297576 8529 raft_consensus.cc:374] T f8016f4a12884d69bc89feaabd86f9fa P 58e38d2ed0094a75a7b48af8719ba90e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:34.297650 8527 ts_tablet_manager.cc:1428] T f8016f4a12884d69bc89feaabd86f9fa P 3ff76b5d25ad47a5897e7bce347ea4bc: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 04:54:34.297670 8529 raft_consensus.cc:729] T f8016f4a12884d69bc89feaabd86f9fa P 58e38d2ed0094a75a7b48af8719ba90e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58e38d2ed0094a75a7b48af8719ba90e, State: Initialized, Role: FOLLOWER
I20260925 04:54:34.297464 8533 tablet_replica.cc:439] T f8016f4a12884d69bc89feaabd86f9fa P 3ff76b5d25ad47a5897e7bce347ea4bc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } } }
I20260925 04:54:34.297840 8533 tablet_replica.cc:442] T f8016f4a12884d69bc89feaabd86f9fa P 3ff76b5d25ad47a5897e7bce347ea4bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:54:34.297886 8529 consensus_queue.cc:260] T f8016f4a12884d69bc89feaabd86f9fa P 58e38d2ed0094a75a7b48af8719ba90e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } }
I20260925 04:54:34.298434 8529 ts_tablet_manager.cc:1428] T f8016f4a12884d69bc89feaabd86f9fa P 58e38d2ed0094a75a7b48af8719ba90e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260925 04:54:34.298532 8528 raft_consensus.cc:348] T f8016f4a12884d69bc89feaabd86f9fa P 0cb0df97fd14448d878f46990713c052 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } }
I20260925 04:54:34.298766 8528 raft_consensus.cc:374] T f8016f4a12884d69bc89feaabd86f9fa P 0cb0df97fd14448d878f46990713c052 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:34.298851 8528 raft_consensus.cc:729] T f8016f4a12884d69bc89feaabd86f9fa P 0cb0df97fd14448d878f46990713c052 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb0df97fd14448d878f46990713c052, State: Initialized, Role: FOLLOWER
I20260925 04:54:34.298761 8535 tablet_replica.cc:439] T f8016f4a12884d69bc89feaabd86f9fa P 58e38d2ed0094a75a7b48af8719ba90e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } } }
I20260925 04:54:34.299094 8535 tablet_replica.cc:442] T f8016f4a12884d69bc89feaabd86f9fa P 58e38d2ed0094a75a7b48af8719ba90e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:54:34.299075 8528 consensus_queue.cc:260] T f8016f4a12884d69bc89feaabd86f9fa P 0cb0df97fd14448d878f46990713c052 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } }
I20260925 04:54:34.299487 8534 tablet_replica.cc:439] T f8016f4a12884d69bc89feaabd86f9fa P 0cb0df97fd14448d878f46990713c052: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } } }
I20260925 04:54:34.299785 8534 tablet_replica.cc:442] T f8016f4a12884d69bc89feaabd86f9fa P 0cb0df97fd14448d878f46990713c052: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:54:34.300374 8528 ts_tablet_manager.cc:1428] T f8016f4a12884d69bc89feaabd86f9fa P 0cb0df97fd14448d878f46990713c052: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260925 04:54:34.453848 8535 raft_consensus.cc:482] T f8016f4a12884d69bc89feaabd86f9fa P 58e38d2ed0094a75a7b48af8719ba90e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:54:34.454166 8535 raft_consensus.cc:504] T f8016f4a12884d69bc89feaabd86f9fa P 58e38d2ed0094a75a7b48af8719ba90e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } }
I20260925 04:54:34.455739 8535 leader_election.cc:288] T f8016f4a12884d69bc89feaabd86f9fa P 58e38d2ed0094a75a7b48af8719ba90e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cb0df97fd14448d878f46990713c052 (127.6.29.129:45705), 3ff76b5d25ad47a5897e7bce347ea4bc (127.6.29.131:37063)
I20260925 04:54:34.463860 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:34.456470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:46923 (local address 127.6.29.129:45705)
0925 04:54:34.456672 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:34.456680 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:34.457348 (+ 668us) server_negotiation.cc:407] Connection header received
0925 04:54:34.457508 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:34.457513 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:34.457605 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:34.457723 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:54:34.458429 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.459286 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.461743 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.462839 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.463011 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:34.463087 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:34.463460 (+ 373us) server_negotiation.cc:299] Negotiation successful
0925 04:54:34.463592 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 04:54:34.464102 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:34.456270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:59029 (local address 127.6.29.131:37063)
0925 04:54:34.457241 (+ 971us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:34.457248 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:34.457270 (+ 22us) server_negotiation.cc:407] Connection header received
0925 04:54:34.457346 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:34.457353 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:34.457433 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:34.457517 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:54:34.458254 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.459460 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.462039 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.463193 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.463351 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:34.463437 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:34.463720 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 04:54:34.463863 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":842}
I20260925 04:54:34.464519 8549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:34.456071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.131:37063 (local address 127.6.29.130:59029)
0925 04:54:34.456781 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:34.456800 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:34.456964 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:34.457527 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:34.457532 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:34.457551 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:54:34.458044 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:34.458062 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.459635 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.459644 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:34.461879 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:34.461894 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.463412 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.463417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:34.463570 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:34.463592 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:54:34.463695 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 04:54:34.463839 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":613,"threads_started":1}
I20260925 04:54:34.464912 8550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:34.456092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:45705 (local address 127.6.29.130:46923)
0925 04:54:34.457240 (+ 1148us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:34.457261 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:34.457378 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:34.457741 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:34.457747 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:34.457776 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:54:34.458258 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:34.458272 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.459481 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.459486 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:34.461599 (+ 2113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:34.461615 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.463073 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.463080 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:34.463268 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:34.463303 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 04:54:34.464378 (+ 1075us) client_negotiation.cc:240] Negotiation successful
0925 04:54:34.464500 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1023,"mutex_wait_us":39,"thread_start_us":878,"threads_started":1}
I20260925 04:54:34.465137 8463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8016f4a12884d69bc89feaabd86f9fa" candidate_uuid: "58e38d2ed0094a75a7b48af8719ba90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" is_pre_election: true
I20260925 04:54:34.465425 8463 raft_consensus.cc:2454] T f8016f4a12884d69bc89feaabd86f9fa P 3ff76b5d25ad47a5897e7bce347ea4bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58e38d2ed0094a75a7b48af8719ba90e in term 0.
I20260925 04:54:34.465893 8252 leader_election.cc:302] T f8016f4a12884d69bc89feaabd86f9fa P 58e38d2ed0094a75a7b48af8719ba90e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ff76b5d25ad47a5897e7bce347ea4bc, 58e38d2ed0094a75a7b48af8719ba90e; no voters:
I20260925 04:54:34.465821 8180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8016f4a12884d69bc89feaabd86f9fa" candidate_uuid: "58e38d2ed0094a75a7b48af8719ba90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb0df97fd14448d878f46990713c052" is_pre_election: true
I20260925 04:54:34.466251 8535 raft_consensus.cc:2781] T f8016f4a12884d69bc89feaabd86f9fa P 58e38d2ed0094a75a7b48af8719ba90e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:54:34.466239 8180 raft_consensus.cc:2454] T f8016f4a12884d69bc89feaabd86f9fa P 0cb0df97fd14448d878f46990713c052 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58e38d2ed0094a75a7b48af8719ba90e in term 0.
I20260925 04:54:34.466378 8535 raft_consensus.cc:482] T f8016f4a12884d69bc89feaabd86f9fa P 58e38d2ed0094a75a7b48af8719ba90e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:54:34.466475 8535 raft_consensus.cc:3037] T f8016f4a12884d69bc89feaabd86f9fa P 58e38d2ed0094a75a7b48af8719ba90e [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:34.469689 8535 raft_consensus.cc:504] T f8016f4a12884d69bc89feaabd86f9fa P 58e38d2ed0094a75a7b48af8719ba90e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } }
I20260925 04:54:34.470307 8535 leader_election.cc:288] T f8016f4a12884d69bc89feaabd86f9fa P 58e38d2ed0094a75a7b48af8719ba90e [CANDIDATE]: Term 1 election: Requested vote from peers 0cb0df97fd14448d878f46990713c052 (127.6.29.129:45705), 3ff76b5d25ad47a5897e7bce347ea4bc (127.6.29.131:37063)
I20260925 04:54:34.470755 8180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8016f4a12884d69bc89feaabd86f9fa" candidate_uuid: "58e38d2ed0094a75a7b48af8719ba90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb0df97fd14448d878f46990713c052"
I20260925 04:54:34.470933 8180 raft_consensus.cc:3037] T f8016f4a12884d69bc89feaabd86f9fa P 0cb0df97fd14448d878f46990713c052 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:34.473928 8180 raft_consensus.cc:2454] T f8016f4a12884d69bc89feaabd86f9fa P 0cb0df97fd14448d878f46990713c052 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58e38d2ed0094a75a7b48af8719ba90e in term 1.
I20260925 04:54:34.474268 8463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8016f4a12884d69bc89feaabd86f9fa" candidate_uuid: "58e38d2ed0094a75a7b48af8719ba90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc"
I20260925 04:54:34.474331 8253 leader_election.cc:302] T f8016f4a12884d69bc89feaabd86f9fa P 58e38d2ed0094a75a7b48af8719ba90e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cb0df97fd14448d878f46990713c052, 58e38d2ed0094a75a7b48af8719ba90e; no voters:
I20260925 04:54:34.474519 8463 raft_consensus.cc:3037] T f8016f4a12884d69bc89feaabd86f9fa P 3ff76b5d25ad47a5897e7bce347ea4bc [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:34.474623 8535 raft_consensus.cc:2781] T f8016f4a12884d69bc89feaabd86f9fa P 58e38d2ed0094a75a7b48af8719ba90e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:54:34.475152 8535 raft_consensus.cc:686] T f8016f4a12884d69bc89feaabd86f9fa P 58e38d2ed0094a75a7b48af8719ba90e [term 1 LEADER]: Becoming Leader. State: Replica: 58e38d2ed0094a75a7b48af8719ba90e, State: Running, Role: LEADER
I20260925 04:54:34.475699 8535 consensus_queue.cc:237] T f8016f4a12884d69bc89feaabd86f9fa P 58e38d2ed0094a75a7b48af8719ba90e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } }
I20260925 04:54:34.476949 8463 raft_consensus.cc:2454] T f8016f4a12884d69bc89feaabd86f9fa P 3ff76b5d25ad47a5897e7bce347ea4bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58e38d2ed0094a75a7b48af8719ba90e in term 1.
I20260925 04:54:34.478901 8551 tablet_replica.cc:439] T f8016f4a12884d69bc89feaabd86f9fa P 58e38d2ed0094a75a7b48af8719ba90e: TxnStatusTablet state changed. Reason: New leader 58e38d2ed0094a75a7b48af8719ba90e. Latest consensus state: current_term: 1 leader_uuid: "58e38d2ed0094a75a7b48af8719ba90e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } } }
I20260925 04:54:34.479280 8551 tablet_replica.cc:442] T f8016f4a12884d69bc89feaabd86f9fa P 58e38d2ed0094a75a7b48af8719ba90e: This TxnStatusTablet replica's current role is: LEADER
I20260925 04:54:34.480026 8553 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 04:54:34.480285 8553 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 04:54:34.480832 8021 catalog_manager.cc:5495] T f8016f4a12884d69bc89feaabd86f9fa P 58e38d2ed0094a75a7b48af8719ba90e reported cstate change: term changed from 0 to 1, leader changed from <none> to 58e38d2ed0094a75a7b48af8719ba90e (127.6.29.130). New cstate: current_term: 1 leader_uuid: "58e38d2ed0094a75a7b48af8719ba90e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } health_report { overall_health: UNKNOWN } } }
I20260925 04:54:34.539695 8545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:34.533653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:40465 (local address 127.0.0.1:57266)
0925 04:54:34.533996 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:34.534012 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:34.534176 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:34.534655 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:34.534660 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:34.534671 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:54:34.534862 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:34.534871 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.535918 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.535927 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:34.537040 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:34.537052 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.537902 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.537908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:34.538113 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:34.538182 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:54:34.539148 (+ 966us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:54:34.539168 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 04:54:34.539235 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 04:54:34.539383 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168}
I20260925 04:54:34.539733 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:34.533679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57266 (local address 127.6.29.130:40465)
0925 04:54:34.533871 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:34.533891 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:34.534087 (+ 196us) server_negotiation.cc:407] Connection header received
0925 04:54:34.534382 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:34.534386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:34.534485 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:34.534636 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:54:34.534990 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.535740 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.537177 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.537736 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.537867 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:34.538400 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:54:34.539033 (+ 633us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:54:34.539197 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:34.539270 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 04:54:34.539364 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 04:54:34.547663 8463 raft_consensus.cc:1261] T f8016f4a12884d69bc89feaabd86f9fa P 3ff76b5d25ad47a5897e7bce347ea4bc [term 1 FOLLOWER]: Refusing update from remote peer 58e38d2ed0094a75a7b48af8719ba90e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:54:34.547698 8180 raft_consensus.cc:1261] T f8016f4a12884d69bc89feaabd86f9fa P 0cb0df97fd14448d878f46990713c052 [term 1 FOLLOWER]: Refusing update from remote peer 58e38d2ed0094a75a7b48af8719ba90e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:54:34.548707 8551 consensus_queue.cc:1035] T f8016f4a12884d69bc89feaabd86f9fa P 58e38d2ed0094a75a7b48af8719ba90e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:54:34.549010 8535 consensus_queue.cc:1035] T f8016f4a12884d69bc89feaabd86f9fa P 58e38d2ed0094a75a7b48af8719ba90e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:54:34.560024 8533 tablet_replica.cc:439] T f8016f4a12884d69bc89feaabd86f9fa P 3ff76b5d25ad47a5897e7bce347ea4bc: TxnStatusTablet state changed. Reason: New leader 58e38d2ed0094a75a7b48af8719ba90e. Latest consensus state: current_term: 1 leader_uuid: "58e38d2ed0094a75a7b48af8719ba90e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } } }
I20260925 04:54:34.560292 8533 tablet_replica.cc:442] T f8016f4a12884d69bc89feaabd86f9fa P 3ff76b5d25ad47a5897e7bce347ea4bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:54:34.562139 8556 mvcc.cc:205] Tried to move back new op lower bound from 7333118257337700352 to 7333118257056423936. Current Snapshot: MvccSnapshot[applied={T|T < 7333118257337700352}]
I20260925 04:54:34.563282 8559 mvcc.cc:205] Tried to move back new op lower bound from 7333118257337700352 to 7333118257056423936. Current Snapshot: MvccSnapshot[applied={T|T < 7333118257337700352}]
I20260925 04:54:34.563539 8534 tablet_replica.cc:439] T f8016f4a12884d69bc89feaabd86f9fa P 0cb0df97fd14448d878f46990713c052: TxnStatusTablet state changed. Reason: New leader 58e38d2ed0094a75a7b48af8719ba90e. Latest consensus state: current_term: 1 leader_uuid: "58e38d2ed0094a75a7b48af8719ba90e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } } }
I20260925 04:54:34.565106 8534 tablet_replica.cc:442] T f8016f4a12884d69bc89feaabd86f9fa P 0cb0df97fd14448d878f46990713c052: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:54:34.567153 8551 tablet_replica.cc:439] T f8016f4a12884d69bc89feaabd86f9fa P 58e38d2ed0094a75a7b48af8719ba90e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "58e38d2ed0094a75a7b48af8719ba90e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } } }
I20260925 04:54:34.567423 8551 tablet_replica.cc:442] T f8016f4a12884d69bc89feaabd86f9fa P 58e38d2ed0094a75a7b48af8719ba90e: This TxnStatusTablet replica's current role is: LEADER
I20260925 04:54:34.567629 8551 tablet_replica.cc:439] T f8016f4a12884d69bc89feaabd86f9fa P 58e38d2ed0094a75a7b48af8719ba90e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "58e38d2ed0094a75a7b48af8719ba90e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } } }
I20260925 04:54:34.567895 8551 tablet_replica.cc:442] T f8016f4a12884d69bc89feaabd86f9fa P 58e38d2ed0094a75a7b48af8719ba90e: This TxnStatusTablet replica's current role is: LEADER
I20260925 04:54:34.570024 8557 mvcc.cc:205] Tried to move back new op lower bound from 7333118257337700352 to 7333118257056423936. Current Snapshot: MvccSnapshot[applied={T|T < 7333118257337700352}]
I20260925 04:54:34.574978 8533 tablet_replica.cc:439] T f8016f4a12884d69bc89feaabd86f9fa P 3ff76b5d25ad47a5897e7bce347ea4bc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "58e38d2ed0094a75a7b48af8719ba90e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } } }
I20260925 04:54:34.575336 8533 tablet_replica.cc:442] T f8016f4a12884d69bc89feaabd86f9fa P 3ff76b5d25ad47a5897e7bce347ea4bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:54:34.577122 8534 tablet_replica.cc:439] T f8016f4a12884d69bc89feaabd86f9fa P 0cb0df97fd14448d878f46990713c052: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "58e38d2ed0094a75a7b48af8719ba90e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } } }
I20260925 04:54:34.577421 8534 tablet_replica.cc:442] T f8016f4a12884d69bc89feaabd86f9fa P 0cb0df97fd14448d878f46990713c052: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:54:34.587811 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:34.580819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59150 (local address 127.6.29.129:45705)
0925 04:54:34.581027 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:34.581037 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:34.581527 (+ 490us) server_negotiation.cc:407] Connection header received
0925 04:54:34.581727 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:34.581730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:34.581817 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:34.581947 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:54:34.582453 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.583192 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.585278 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.585793 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.585938 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:34.586444 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:54:34.587101 (+ 657us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:54:34.587261 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:34.587390 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 04:54:34.587453 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 04:54:34.596844 8534 consensus_queue.cc:1035] T 7ae3064d66a74841887816fb83f6e557 P 0cb0df97fd14448d878f46990713c052 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:54:34.603006 8538 consensus_queue.cc:1035] T 7ae3064d66a74841887816fb83f6e557 P 0cb0df97fd14448d878f46990713c052 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:54:34.619407 8570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:34.599454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:40465 (local address 127.6.29.129:56565)
0925 04:54:34.613705 (+ 14251us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:34.613728 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:34.613971 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:34.614374 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:34.614377 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:34.614386 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:54:34.614627 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:34.614634 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.616096 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.616101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:34.617281 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:34.617294 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.618272 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.618276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:34.618428 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:34.618487 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:54:34.618944 (+ 457us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:54:34.618951 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 04:54:34.619010 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 04:54:34.619111 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14065,"thread_start_us":14221,"threads_started":1}
I20260925 04:54:34.619426 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:34.599898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:56565 (local address 127.6.29.130:40465)
0925 04:54:34.600117 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:34.600126 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:34.613819 (+ 13693us) server_negotiation.cc:407] Connection header received
0925 04:54:34.614130 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:34.614136 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:34.614245 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:34.614339 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:54:34.614763 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.615953 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.617452 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.618118 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.618260 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:34.618714 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:54:34.618849 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:54:34.618938 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:34.619055 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 04:54:34.619134 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 04:54:34.662477 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:34.655556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:50923 (local address 127.6.29.129:45705)
0925 04:54:34.655806 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:34.655812 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:34.656315 (+ 503us) server_negotiation.cc:407] Connection header received
0925 04:54:34.656504 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:34.656509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:34.656590 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:34.656688 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:54:34.657335 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.658280 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.660618 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.661267 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.661419 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:34.661891 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:54:34.661984 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:54:34.662051 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:34.662197 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 04:54:34.662255 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 04:54:34.662743 8575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:34.655498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:45705 (local address 127.6.29.130:50923)
0925 04:54:34.656196 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:34.656217 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:34.656368 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:34.656712 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:34.656717 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:34.656729 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:54:34.657148 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:34.657158 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.658415 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.658421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:34.660475 (+ 2054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:34.660490 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.661449 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.661455 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:34.661643 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:34.661722 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:54:34.662079 (+ 357us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:54:34.662090 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 04:54:34.662177 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 04:54:34.662335 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":478,"threads_started":1}
I20260925 04:54:34.706704 8021 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:54:34.707003 8021 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:54:34.711088 8021 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3473cd541f7042a7903a56c2aa871845: Sending DeleteTablet for 3 replicas of tablet 7ae3064d66a74841887816fb83f6e557
I20260925 04:54:34.712361 8160 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ae3064d66a74841887816fb83f6e557 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:54:34 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:59142
I20260925 04:54:34.712396 8299 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ae3064d66a74841887816fb83f6e557 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:54:34 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:57260
I20260925 04:54:34.713382 8438 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ae3064d66a74841887816fb83f6e557 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:54:34 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:39966
I20260925 04:54:34.720410 8021 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:54:34.721058 8021 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:54:34.723666 8578 tablet_replica.cc:330] T 7ae3064d66a74841887816fb83f6e557 P 0cb0df97fd14448d878f46990713c052: stopping tablet replica
I20260925 04:54:34.724314 8579 tablet_replica.cc:330] T 7ae3064d66a74841887816fb83f6e557 P 58e38d2ed0094a75a7b48af8719ba90e: stopping tablet replica
I20260925 04:54:34.727684 8578 raft_consensus.cc:2229] T 7ae3064d66a74841887816fb83f6e557 P 0cb0df97fd14448d878f46990713c052 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:54:34.727639 8579 raft_consensus.cc:2229] T 7ae3064d66a74841887816fb83f6e557 P 58e38d2ed0094a75a7b48af8719ba90e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:54:34.729106 8579 raft_consensus.cc:2258] T 7ae3064d66a74841887816fb83f6e557 P 58e38d2ed0094a75a7b48af8719ba90e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:54:34.729640 8578 raft_consensus.cc:2258] T 7ae3064d66a74841887816fb83f6e557 P 0cb0df97fd14448d878f46990713c052 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:54:34.731015 8299 tablet_service.cc:1467] Processing CreateTablet for tablet f6ecd73d8f18424d8dccf341a9fb08b4 (DEFAULT_TABLE table=test-table [id=4ce7f179f3d7470081f8c1bd2f04d079]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:54:34.731284 8160 tablet_service.cc:1467] Processing CreateTablet for tablet f6ecd73d8f18424d8dccf341a9fb08b4 (DEFAULT_TABLE table=test-table [id=4ce7f179f3d7470081f8c1bd2f04d079]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:54:34.731577 8437 tablet_service.cc:1467] Processing CreateTablet for tablet f6ecd73d8f18424d8dccf341a9fb08b4 (DEFAULT_TABLE table=test-table [id=4ce7f179f3d7470081f8c1bd2f04d079]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:54:34.732129 8437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6ecd73d8f18424d8dccf341a9fb08b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:54:34.732237 8160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6ecd73d8f18424d8dccf341a9fb08b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:54:34.732272 8299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6ecd73d8f18424d8dccf341a9fb08b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:54:34.733709 8579 ts_tablet_manager.cc:1905] T 7ae3064d66a74841887816fb83f6e557 P 58e38d2ed0094a75a7b48af8719ba90e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:54:34.738719 8529 tablet_bootstrap.cc:492] T f6ecd73d8f18424d8dccf341a9fb08b4 P 58e38d2ed0094a75a7b48af8719ba90e: Bootstrap starting.
I20260925 04:54:34.740554 8529 tablet_bootstrap.cc:654] T f6ecd73d8f18424d8dccf341a9fb08b4 P 58e38d2ed0094a75a7b48af8719ba90e: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:34.742782 8528 tablet_bootstrap.cc:492] T f6ecd73d8f18424d8dccf341a9fb08b4 P 0cb0df97fd14448d878f46990713c052: Bootstrap starting.
I20260925 04:54:34.744896 8528 tablet_bootstrap.cc:654] T f6ecd73d8f18424d8dccf341a9fb08b4 P 0cb0df97fd14448d878f46990713c052: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:34.745048 8578 ts_tablet_manager.cc:1905] T 7ae3064d66a74841887816fb83f6e557 P 0cb0df97fd14448d878f46990713c052: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:54:34.748226 8527 tablet_bootstrap.cc:492] T f6ecd73d8f18424d8dccf341a9fb08b4 P 3ff76b5d25ad47a5897e7bce347ea4bc: Bootstrap starting.
I20260925 04:54:34.749887 8527 tablet_bootstrap.cc:654] T f6ecd73d8f18424d8dccf341a9fb08b4 P 3ff76b5d25ad47a5897e7bce347ea4bc: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:34.751652 8579 ts_tablet_manager.cc:1918] T 7ae3064d66a74841887816fb83f6e557 P 58e38d2ed0094a75a7b48af8719ba90e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:54:34.751860 8579 log.cc:1196] T 7ae3064d66a74841887816fb83f6e557 P 58e38d2ed0094a75a7b48af8719ba90e: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-1/wal/wals/7ae3064d66a74841887816fb83f6e557
I20260925 04:54:34.752391 8579 ts_tablet_manager.cc:1939] T 7ae3064d66a74841887816fb83f6e557 P 58e38d2ed0094a75a7b48af8719ba90e: Deleting consensus metadata
I20260925 04:54:34.754418 8007 catalog_manager.cc:4841] TS 58e38d2ed0094a75a7b48af8719ba90e (127.6.29.130:40465): tablet 7ae3064d66a74841887816fb83f6e557 (table test-table [id=080df66c52c74ed4b083745f65f39623]) successfully deleted
I20260925 04:54:34.759088 8529 tablet_bootstrap.cc:492] T f6ecd73d8f18424d8dccf341a9fb08b4 P 58e38d2ed0094a75a7b48af8719ba90e: No bootstrap required, opened a new log
I20260925 04:54:34.759268 8529 ts_tablet_manager.cc:1397] T f6ecd73d8f18424d8dccf341a9fb08b4 P 58e38d2ed0094a75a7b48af8719ba90e: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.008s
I20260925 04:54:34.760130 8528 tablet_bootstrap.cc:492] T f6ecd73d8f18424d8dccf341a9fb08b4 P 0cb0df97fd14448d878f46990713c052: No bootstrap required, opened a new log
I20260925 04:54:34.760203 8529 raft_consensus.cc:348] T f6ecd73d8f18424d8dccf341a9fb08b4 P 58e38d2ed0094a75a7b48af8719ba90e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } }
I20260925 04:54:34.760380 8528 ts_tablet_manager.cc:1397] T f6ecd73d8f18424d8dccf341a9fb08b4 P 0cb0df97fd14448d878f46990713c052: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.004s
I20260925 04:54:34.760473 8529 raft_consensus.cc:374] T f6ecd73d8f18424d8dccf341a9fb08b4 P 58e38d2ed0094a75a7b48af8719ba90e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:34.760550 8529 raft_consensus.cc:729] T f6ecd73d8f18424d8dccf341a9fb08b4 P 58e38d2ed0094a75a7b48af8719ba90e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58e38d2ed0094a75a7b48af8719ba90e, State: Initialized, Role: FOLLOWER
I20260925 04:54:34.760830 8529 consensus_queue.cc:260] T f6ecd73d8f18424d8dccf341a9fb08b4 P 58e38d2ed0094a75a7b48af8719ba90e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } }
I20260925 04:54:34.761441 8529 ts_tablet_manager.cc:1428] T f6ecd73d8f18424d8dccf341a9fb08b4 P 58e38d2ed0094a75a7b48af8719ba90e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 04:54:34.762027 8528 raft_consensus.cc:348] T f6ecd73d8f18424d8dccf341a9fb08b4 P 0cb0df97fd14448d878f46990713c052 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } }
I20260925 04:54:34.762275 8528 raft_consensus.cc:374] T f6ecd73d8f18424d8dccf341a9fb08b4 P 0cb0df97fd14448d878f46990713c052 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:34.762384 8528 raft_consensus.cc:729] T f6ecd73d8f18424d8dccf341a9fb08b4 P 0cb0df97fd14448d878f46990713c052 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb0df97fd14448d878f46990713c052, State: Initialized, Role: FOLLOWER
I20260925 04:54:34.762668 8528 consensus_queue.cc:260] T f6ecd73d8f18424d8dccf341a9fb08b4 P 0cb0df97fd14448d878f46990713c052 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } }
I20260925 04:54:34.763288 8528 ts_tablet_manager.cc:1428] T f6ecd73d8f18424d8dccf341a9fb08b4 P 0cb0df97fd14448d878f46990713c052: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260925 04:54:34.774339 8527 tablet_bootstrap.cc:492] T f6ecd73d8f18424d8dccf341a9fb08b4 P 3ff76b5d25ad47a5897e7bce347ea4bc: No bootstrap required, opened a new log
I20260925 04:54:34.774559 8527 ts_tablet_manager.cc:1397] T f6ecd73d8f18424d8dccf341a9fb08b4 P 3ff76b5d25ad47a5897e7bce347ea4bc: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.004s
I20260925 04:54:34.775067 8580 tablet_replica.cc:330] T 7ae3064d66a74841887816fb83f6e557 P 3ff76b5d25ad47a5897e7bce347ea4bc: stopping tablet replica
I20260925 04:54:34.775576 8580 raft_consensus.cc:2229] T 7ae3064d66a74841887816fb83f6e557 P 3ff76b5d25ad47a5897e7bce347ea4bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:54:34.777132 8578 ts_tablet_manager.cc:1918] T 7ae3064d66a74841887816fb83f6e557 P 0cb0df97fd14448d878f46990713c052: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:54:34.777346 8578 log.cc:1196] T 7ae3064d66a74841887816fb83f6e557 P 0cb0df97fd14448d878f46990713c052: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-0/wal/wals/7ae3064d66a74841887816fb83f6e557
I20260925 04:54:34.775481 8527 raft_consensus.cc:348] T f6ecd73d8f18424d8dccf341a9fb08b4 P 3ff76b5d25ad47a5897e7bce347ea4bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } }
I20260925 04:54:34.777562 8527 raft_consensus.cc:374] T f6ecd73d8f18424d8dccf341a9fb08b4 P 3ff76b5d25ad47a5897e7bce347ea4bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:34.777711 8527 raft_consensus.cc:729] T f6ecd73d8f18424d8dccf341a9fb08b4 P 3ff76b5d25ad47a5897e7bce347ea4bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ff76b5d25ad47a5897e7bce347ea4bc, State: Initialized, Role: FOLLOWER
I20260925 04:54:34.777951 8578 ts_tablet_manager.cc:1939] T 7ae3064d66a74841887816fb83f6e557 P 0cb0df97fd14448d878f46990713c052: Deleting consensus metadata
I20260925 04:54:34.778118 8527 consensus_queue.cc:260] T f6ecd73d8f18424d8dccf341a9fb08b4 P 3ff76b5d25ad47a5897e7bce347ea4bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } }
I20260925 04:54:34.779726 8010 catalog_manager.cc:4841] TS 0cb0df97fd14448d878f46990713c052 (127.6.29.129:45705): tablet 7ae3064d66a74841887816fb83f6e557 (table test-table [id=080df66c52c74ed4b083745f65f39623]) successfully deleted
I20260925 04:54:34.780064 8527 ts_tablet_manager.cc:1428] T f6ecd73d8f18424d8dccf341a9fb08b4 P 3ff76b5d25ad47a5897e7bce347ea4bc: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20260925 04:54:34.778681 8580 raft_consensus.cc:2258] T 7ae3064d66a74841887816fb83f6e557 P 3ff76b5d25ad47a5897e7bce347ea4bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:54:34.782635 8580 ts_tablet_manager.cc:1905] T 7ae3064d66a74841887816fb83f6e557 P 3ff76b5d25ad47a5897e7bce347ea4bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:54:34.790160 8580 ts_tablet_manager.cc:1918] T 7ae3064d66a74841887816fb83f6e557 P 3ff76b5d25ad47a5897e7bce347ea4bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:54:34.790375 8580 log.cc:1196] T 7ae3064d66a74841887816fb83f6e557 P 3ff76b5d25ad47a5897e7bce347ea4bc: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-2/wal/wals/7ae3064d66a74841887816fb83f6e557
I20260925 04:54:34.790892 8580 ts_tablet_manager.cc:1939] T 7ae3064d66a74841887816fb83f6e557 P 3ff76b5d25ad47a5897e7bce347ea4bc: Deleting consensus metadata
I20260925 04:54:34.792873 8009 catalog_manager.cc:4841] TS 3ff76b5d25ad47a5897e7bce347ea4bc (127.6.29.131:37063): tablet 7ae3064d66a74841887816fb83f6e557 (table test-table [id=080df66c52c74ed4b083745f65f39623]) successfully deleted
I20260925 04:54:34.845125 8533 raft_consensus.cc:482] T f6ecd73d8f18424d8dccf341a9fb08b4 P 3ff76b5d25ad47a5897e7bce347ea4bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:54:34.845358 8533 raft_consensus.cc:504] T f6ecd73d8f18424d8dccf341a9fb08b4 P 3ff76b5d25ad47a5897e7bce347ea4bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } }
I20260925 04:54:34.847286 8533 leader_election.cc:288] T f6ecd73d8f18424d8dccf341a9fb08b4 P 3ff76b5d25ad47a5897e7bce347ea4bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cb0df97fd14448d878f46990713c052 (127.6.29.129:45705), 58e38d2ed0094a75a7b48af8719ba90e (127.6.29.130:40465)
I20260925 04:54:34.868674 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:34.848402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:55743 (local address 127.6.29.130:40465)
0925 04:54:34.848565 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:34.848571 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:34.860377 (+ 11806us) server_negotiation.cc:407] Connection header received
0925 04:54:34.860664 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:34.860672 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:34.860768 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:34.860845 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:54:34.861704 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.862903 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.865920 (+ 3017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.867293 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.867444 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:34.867557 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:34.868214 (+ 657us) server_negotiation.cc:299] Negotiation successful
0925 04:54:34.868363 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 04:54:34.868791 8523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:34.847925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:42465 (local address 127.6.29.129:45705)
0925 04:54:34.848161 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:34.848168 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:34.860379 (+ 12211us) server_negotiation.cc:407] Connection header received
0925 04:54:34.860797 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:34.860803 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:34.860884 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:34.860952 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:54:34.861977 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.863184 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.865919 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.867313 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.867462 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:34.867552 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:34.868292 (+ 740us) server_negotiation.cc:299] Negotiation successful
0925 04:54:34.868446 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 04:54:34.869702 8584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:34.847620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:45705 (local address 127.6.29.131:42465)
0925 04:54:34.860261 (+ 12641us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:34.860282 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:34.860547 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:34.861159 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:34.861165 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:34.861210 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:54:34.861760 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:34.861775 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.863370 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.863377 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:34.865721 (+ 2344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:34.865737 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.867847 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.867857 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:34.868105 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:34.868136 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 04:54:34.869122 (+ 986us) client_negotiation.cc:240] Negotiation successful
0925 04:54:34.869271 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12061,"mutex_wait_us":70,"thread_start_us":12308,"threads_started":1}
I20260925 04:54:34.870023 8585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:34.847969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:40465 (local address 127.6.29.131:55743)
0925 04:54:34.860255 (+ 12286us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:34.860279 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:34.860501 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:34.861069 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:34.861074 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:34.861101 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:54:34.861520 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:34.861532 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.863091 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.863096 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:34.865730 (+ 2634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:34.865744 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.867743 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.867749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:34.867995 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:34.868032 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 04:54:34.869254 (+ 1222us) client_negotiation.cc:240] Negotiation successful
0925 04:54:34.869400 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12111,"mutex_wait_us":48,"thread_start_us":11578,"threads_started":1}
I20260925 04:54:34.870393 8180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecd73d8f18424d8dccf341a9fb08b4" candidate_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb0df97fd14448d878f46990713c052" is_pre_election: true
I20260925 04:54:34.870702 8180 raft_consensus.cc:2454] T f6ecd73d8f18424d8dccf341a9fb08b4 P 0cb0df97fd14448d878f46990713c052 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ff76b5d25ad47a5897e7bce347ea4bc in term 0.
I20260925 04:54:34.870968 8319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecd73d8f18424d8dccf341a9fb08b4" candidate_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58e38d2ed0094a75a7b48af8719ba90e" is_pre_election: true
I20260925 04:54:34.871208 8319 raft_consensus.cc:2454] T f6ecd73d8f18424d8dccf341a9fb08b4 P 58e38d2ed0094a75a7b48af8719ba90e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ff76b5d25ad47a5897e7bce347ea4bc in term 0.
I20260925 04:54:34.871289 8392 leader_election.cc:302] T f6ecd73d8f18424d8dccf341a9fb08b4 P 3ff76b5d25ad47a5897e7bce347ea4bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cb0df97fd14448d878f46990713c052, 3ff76b5d25ad47a5897e7bce347ea4bc; no voters:
I20260925 04:54:34.871814 8533 raft_consensus.cc:2781] T f6ecd73d8f18424d8dccf341a9fb08b4 P 3ff76b5d25ad47a5897e7bce347ea4bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:54:34.871948 8533 raft_consensus.cc:482] T f6ecd73d8f18424d8dccf341a9fb08b4 P 3ff76b5d25ad47a5897e7bce347ea4bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:54:34.872061 8533 raft_consensus.cc:3037] T f6ecd73d8f18424d8dccf341a9fb08b4 P 3ff76b5d25ad47a5897e7bce347ea4bc [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:34.875243 8533 raft_consensus.cc:504] T f6ecd73d8f18424d8dccf341a9fb08b4 P 3ff76b5d25ad47a5897e7bce347ea4bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } }
I20260925 04:54:34.876456 8319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecd73d8f18424d8dccf341a9fb08b4" candidate_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58e38d2ed0094a75a7b48af8719ba90e"
I20260925 04:54:34.876487 8180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6ecd73d8f18424d8dccf341a9fb08b4" candidate_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb0df97fd14448d878f46990713c052"
I20260925 04:54:34.876682 8180 raft_consensus.cc:3037] T f6ecd73d8f18424d8dccf341a9fb08b4 P 0cb0df97fd14448d878f46990713c052 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:34.876718 8319 raft_consensus.cc:3037] T f6ecd73d8f18424d8dccf341a9fb08b4 P 58e38d2ed0094a75a7b48af8719ba90e [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:34.879935 8180 raft_consensus.cc:2454] T f6ecd73d8f18424d8dccf341a9fb08b4 P 0cb0df97fd14448d878f46990713c052 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ff76b5d25ad47a5897e7bce347ea4bc in term 1.
I20260925 04:54:34.879972 8319 raft_consensus.cc:2454] T f6ecd73d8f18424d8dccf341a9fb08b4 P 58e38d2ed0094a75a7b48af8719ba90e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ff76b5d25ad47a5897e7bce347ea4bc in term 1.
I20260925 04:54:34.880470 8392 leader_election.cc:302] T f6ecd73d8f18424d8dccf341a9fb08b4 P 3ff76b5d25ad47a5897e7bce347ea4bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cb0df97fd14448d878f46990713c052, 3ff76b5d25ad47a5897e7bce347ea4bc; no voters:
I20260925 04:54:34.881940 8533 leader_election.cc:288] T f6ecd73d8f18424d8dccf341a9fb08b4 P 3ff76b5d25ad47a5897e7bce347ea4bc [CANDIDATE]: Term 1 election: Requested vote from peers 0cb0df97fd14448d878f46990713c052 (127.6.29.129:45705), 58e38d2ed0094a75a7b48af8719ba90e (127.6.29.130:40465)
I20260925 04:54:34.882170 8533 raft_consensus.cc:2781] T f6ecd73d8f18424d8dccf341a9fb08b4 P 3ff76b5d25ad47a5897e7bce347ea4bc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:54:34.882323 8533 raft_consensus.cc:686] T f6ecd73d8f18424d8dccf341a9fb08b4 P 3ff76b5d25ad47a5897e7bce347ea4bc [term 1 LEADER]: Becoming Leader. State: Replica: 3ff76b5d25ad47a5897e7bce347ea4bc, State: Running, Role: LEADER
I20260925 04:54:34.882777 8533 consensus_queue.cc:237] T f6ecd73d8f18424d8dccf341a9fb08b4 P 3ff76b5d25ad47a5897e7bce347ea4bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } }
I20260925 04:54:34.891546 8023 catalog_manager.cc:5495] T f6ecd73d8f18424d8dccf341a9fb08b4 P 3ff76b5d25ad47a5897e7bce347ea4bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ff76b5d25ad47a5897e7bce347ea4bc (127.6.29.131). New cstate: current_term: 1 leader_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } health_report { overall_health: HEALTHY } } }
I20260925 04:54:34.935259 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:34.928130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39968 (local address 127.6.29.131:37063)
0925 04:54:34.928275 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:34.928281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:34.928297 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:54:34.928388 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:34.928393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:34.928491 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:34.928637 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:54:34.929160 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.930202 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.932109 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.932777 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.932886 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:34.933435 (+ 549us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:54:34.934177 (+ 742us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:54:34.934321 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:34.934891 (+ 570us) server_negotiation.cc:299] Negotiation successful
0925 04:54:34.934977 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 04:54:34.979939 8526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:34.967888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:37841 (local address 127.6.29.130:40465)
0925 04:54:34.968071 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:34.968082 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:34.973143 (+ 5061us) server_negotiation.cc:407] Connection header received
0925 04:54:34.973462 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:34.973468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:34.973554 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:34.973616 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:54:34.974392 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.975482 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.977631 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.978293 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.978389 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:34.979118 (+ 729us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:54:34.979262 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:54:34.979322 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:34.979532 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 04:54:34.979615 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 04:54:34.980962 8590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:34.967906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:40465 (local address 127.6.29.131:37841)
0925 04:54:34.973020 (+ 5114us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:34.973043 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:34.973281 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:34.973806 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:34.973811 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:34.973835 (+ 24us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:54:34.974182 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:34.974194 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.975664 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.975669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:34.977471 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:34.977486 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:34.978582 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:34.978588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:34.978826 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:34.978904 (+ 78us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:54:34.979401 (+ 497us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:54:34.979410 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 04:54:34.980395 (+ 985us) client_negotiation.cc:240] Negotiation successful
0925 04:54:34.980540 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4925,"thread_start_us":11855,"threads_started":1}
I20260925 04:54:34.993889 8180 raft_consensus.cc:1261] T f6ecd73d8f18424d8dccf341a9fb08b4 P 0cb0df97fd14448d878f46990713c052 [term 1 FOLLOWER]: Refusing update from remote peer 3ff76b5d25ad47a5897e7bce347ea4bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:54:34.993947 8319 raft_consensus.cc:1261] T f6ecd73d8f18424d8dccf341a9fb08b4 P 58e38d2ed0094a75a7b48af8719ba90e [term 1 FOLLOWER]: Refusing update from remote peer 3ff76b5d25ad47a5897e7bce347ea4bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:54:34.994577 8533 consensus_queue.cc:1035] T f6ecd73d8f18424d8dccf341a9fb08b4 P 3ff76b5d25ad47a5897e7bce347ea4bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:54:34.995162 8586 consensus_queue.cc:1035] T f6ecd73d8f18424d8dccf341a9fb08b4 P 3ff76b5d25ad47a5897e7bce347ea4bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:54:35.040025 8575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:35.034132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.131:37063 (local address 127.6.29.130:53557)
0925 04:54:35.034342 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:35.034360 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:35.034444 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:35.034825 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:35.034829 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:35.034840 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:54:35.035014 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:35.035022 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:35.036481 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:35.036485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:35.037915 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:35.037946 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:35.038782 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:35.038788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:35.038950 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:35.038995 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:54:35.039463 (+ 468us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:54:35.039475 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 04:54:35.039548 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 04:54:35.039680 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260925 04:54:35.040863 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:35.034171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:53557 (local address 127.6.29.131:37063)
0925 04:54:35.034596 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:35.034601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:35.034615 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:54:35.034658 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:35.034660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:35.034713 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:35.034810 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:54:35.035184 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:35.036316 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:35.038084 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:35.038627 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:35.038786 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:35.039174 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:54:35.039323 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:54:35.039416 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:35.040376 (+ 960us) server_negotiation.cc:299] Negotiation successful
0925 04:54:35.040452 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341}
W20260925 04:54:35.042311 8559 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 04:54:36.655270 8023 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:54:36.655534 8023 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:54:36.659973 8023 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3473cd541f7042a7903a56c2aa871845: Sending DeleteTablet for 3 replicas of tablet f6ecd73d8f18424d8dccf341a9fb08b4
I20260925 04:54:36.660983 8299 tablet_service.cc:1514] Processing DeleteTablet for tablet f6ecd73d8f18424d8dccf341a9fb08b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:54:36 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:57260
I20260925 04:54:36.661005 8160 tablet_service.cc:1514] Processing DeleteTablet for tablet f6ecd73d8f18424d8dccf341a9fb08b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:54:36 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:59142
I20260925 04:54:36.661064 8438 tablet_service.cc:1514] Processing DeleteTablet for tablet f6ecd73d8f18424d8dccf341a9fb08b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:54:36 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:39966
I20260925 04:54:36.672014 8620 tablet_replica.cc:330] T f6ecd73d8f18424d8dccf341a9fb08b4 P 3ff76b5d25ad47a5897e7bce347ea4bc: stopping tablet replica
I20260925 04:54:36.672223 8618 tablet_replica.cc:330] T f6ecd73d8f18424d8dccf341a9fb08b4 P 58e38d2ed0094a75a7b48af8719ba90e: stopping tablet replica
I20260925 04:54:36.672405 8620 raft_consensus.cc:2229] T f6ecd73d8f18424d8dccf341a9fb08b4 P 3ff76b5d25ad47a5897e7bce347ea4bc [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:54:36.672458 8618 raft_consensus.cc:2229] T f6ecd73d8f18424d8dccf341a9fb08b4 P 58e38d2ed0094a75a7b48af8719ba90e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:54:36.672717 8618 raft_consensus.cc:2258] T f6ecd73d8f18424d8dccf341a9fb08b4 P 58e38d2ed0094a75a7b48af8719ba90e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:54:36.673180 8620 raft_consensus.cc:2258] T f6ecd73d8f18424d8dccf341a9fb08b4 P 3ff76b5d25ad47a5897e7bce347ea4bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:54:36.673840 8618 ts_tablet_manager.cc:1905] T f6ecd73d8f18424d8dccf341a9fb08b4 P 58e38d2ed0094a75a7b48af8719ba90e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:54:36.674171 8620 ts_tablet_manager.cc:1905] T f6ecd73d8f18424d8dccf341a9fb08b4 P 3ff76b5d25ad47a5897e7bce347ea4bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:54:36.677423 8619 tablet_replica.cc:330] T f6ecd73d8f18424d8dccf341a9fb08b4 P 0cb0df97fd14448d878f46990713c052: stopping tablet replica
I20260925 04:54:36.677788 8619 raft_consensus.cc:2229] T f6ecd73d8f18424d8dccf341a9fb08b4 P 0cb0df97fd14448d878f46990713c052 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:54:36.678089 8619 raft_consensus.cc:2258] T f6ecd73d8f18424d8dccf341a9fb08b4 P 0cb0df97fd14448d878f46990713c052 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:54:36.679167 8619 ts_tablet_manager.cc:1905] T f6ecd73d8f18424d8dccf341a9fb08b4 P 0cb0df97fd14448d878f46990713c052: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:54:36.680789 8620 ts_tablet_manager.cc:1918] T f6ecd73d8f18424d8dccf341a9fb08b4 P 3ff76b5d25ad47a5897e7bce347ea4bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 04:54:36.680789 8618 ts_tablet_manager.cc:1918] T f6ecd73d8f18424d8dccf341a9fb08b4 P 58e38d2ed0094a75a7b48af8719ba90e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 04:54:36.680932 8618 log.cc:1196] T f6ecd73d8f18424d8dccf341a9fb08b4 P 58e38d2ed0094a75a7b48af8719ba90e: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-1/wal/wals/f6ecd73d8f18424d8dccf341a9fb08b4
I20260925 04:54:36.680994 8620 log.cc:1196] T f6ecd73d8f18424d8dccf341a9fb08b4 P 3ff76b5d25ad47a5897e7bce347ea4bc: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-2/wal/wals/f6ecd73d8f18424d8dccf341a9fb08b4
I20260925 04:54:36.681476 8618 ts_tablet_manager.cc:1939] T f6ecd73d8f18424d8dccf341a9fb08b4 P 58e38d2ed0094a75a7b48af8719ba90e: Deleting consensus metadata
I20260925 04:54:36.681530 8620 ts_tablet_manager.cc:1939] T f6ecd73d8f18424d8dccf341a9fb08b4 P 3ff76b5d25ad47a5897e7bce347ea4bc: Deleting consensus metadata
I20260925 04:54:36.682651 8009 catalog_manager.cc:4841] TS 3ff76b5d25ad47a5897e7bce347ea4bc (127.6.29.131:37063): tablet f6ecd73d8f18424d8dccf341a9fb08b4 (table test-table [id=4ce7f179f3d7470081f8c1bd2f04d079]) successfully deleted
I20260925 04:54:36.682687 8007 catalog_manager.cc:4841] TS 58e38d2ed0094a75a7b48af8719ba90e (127.6.29.130:40465): tablet f6ecd73d8f18424d8dccf341a9fb08b4 (table test-table [id=4ce7f179f3d7470081f8c1bd2f04d079]) successfully deleted
I20260925 04:54:36.686450 8619 ts_tablet_manager.cc:1918] T f6ecd73d8f18424d8dccf341a9fb08b4 P 0cb0df97fd14448d878f46990713c052: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 04:54:36.686592 8619 log.cc:1196] T f6ecd73d8f18424d8dccf341a9fb08b4 P 0cb0df97fd14448d878f46990713c052: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-0/wal/wals/f6ecd73d8f18424d8dccf341a9fb08b4
I20260925 04:54:36.687098 8619 ts_tablet_manager.cc:1939] T f6ecd73d8f18424d8dccf341a9fb08b4 P 0cb0df97fd14448d878f46990713c052: Deleting consensus metadata
I20260925 04:54:36.688212 8010 catalog_manager.cc:4841] TS 0cb0df97fd14448d878f46990713c052 (127.6.29.129:45705): tablet f6ecd73d8f18424d8dccf341a9fb08b4 (table test-table [id=4ce7f179f3d7470081f8c1bd2f04d079]) successfully deleted
Sep 25 04:54:36 dist-test-slave-xzjr krb5kdc[7978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312076, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 04:54:36 dist-test-slave-xzjr krb5kdc[7978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312076, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:54:36.718969 8628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:36.705267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42608 (local address 127.6.29.190:34435)
0925 04:54:36.705775 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:36.705779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:36.705841 (+ 62us) server_negotiation.cc:407] Connection header received
0925 04:54:36.705992 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:36.705996 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:36.706056 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:36.706154 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:36.707384 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:36.708284 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:36.709497 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:36.709894 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:36.709985 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:36.713912 (+ 3927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:36.713916 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:36.713920 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:36.713950 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:36.716692 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:36.717336 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:36.717340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:36.717342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:36.717467 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:36.717830 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:36.717833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:36.717835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:36.718082 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:36.718252 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:36.718541 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 04:54:36.718713 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":278,"threads_started":1}
I20260925 04:54:36.722630 8023 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42608:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:54:36.723435 8023 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:54:36.729830 8160 tablet_service.cc:1467] Processing CreateTablet for tablet 16b46048dfd94f3cb5c7825a366687bb (DEFAULT_TABLE table=test-table [id=bb3577e06aca4ad282d6b6db4f2eb885]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:54:36.729972 8299 tablet_service.cc:1467] Processing CreateTablet for tablet 16b46048dfd94f3cb5c7825a366687bb (DEFAULT_TABLE table=test-table [id=bb3577e06aca4ad282d6b6db4f2eb885]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:54:36.730134 8438 tablet_service.cc:1467] Processing CreateTablet for tablet 16b46048dfd94f3cb5c7825a366687bb (DEFAULT_TABLE table=test-table [id=bb3577e06aca4ad282d6b6db4f2eb885]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:54:36.730273 8160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16b46048dfd94f3cb5c7825a366687bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:54:36.730343 8299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16b46048dfd94f3cb5c7825a366687bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:54:36.730568 8438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16b46048dfd94f3cb5c7825a366687bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:54:36.736248 8629 tablet_bootstrap.cc:492] T 16b46048dfd94f3cb5c7825a366687bb P 3ff76b5d25ad47a5897e7bce347ea4bc: Bootstrap starting.
I20260925 04:54:36.737613 8630 tablet_bootstrap.cc:492] T 16b46048dfd94f3cb5c7825a366687bb P 58e38d2ed0094a75a7b48af8719ba90e: Bootstrap starting.
I20260925 04:54:36.737195 8631 tablet_bootstrap.cc:492] T 16b46048dfd94f3cb5c7825a366687bb P 0cb0df97fd14448d878f46990713c052: Bootstrap starting.
I20260925 04:54:36.738137 8629 tablet_bootstrap.cc:654] T 16b46048dfd94f3cb5c7825a366687bb P 3ff76b5d25ad47a5897e7bce347ea4bc: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:36.739434 8630 tablet_bootstrap.cc:654] T 16b46048dfd94f3cb5c7825a366687bb P 58e38d2ed0094a75a7b48af8719ba90e: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:36.739634 8631 tablet_bootstrap.cc:654] T 16b46048dfd94f3cb5c7825a366687bb P 0cb0df97fd14448d878f46990713c052: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:36.741487 8631 tablet_bootstrap.cc:492] T 16b46048dfd94f3cb5c7825a366687bb P 0cb0df97fd14448d878f46990713c052: No bootstrap required, opened a new log
I20260925 04:54:36.741647 8631 ts_tablet_manager.cc:1397] T 16b46048dfd94f3cb5c7825a366687bb P 0cb0df97fd14448d878f46990713c052: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 04:54:36.741703 8630 tablet_bootstrap.cc:492] T 16b46048dfd94f3cb5c7825a366687bb P 58e38d2ed0094a75a7b48af8719ba90e: No bootstrap required, opened a new log
I20260925 04:54:36.741797 8630 ts_tablet_manager.cc:1397] T 16b46048dfd94f3cb5c7825a366687bb P 58e38d2ed0094a75a7b48af8719ba90e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 04:54:36.742311 8631 raft_consensus.cc:348] T 16b46048dfd94f3cb5c7825a366687bb P 0cb0df97fd14448d878f46990713c052 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } }
I20260925 04:54:36.742343 8630 raft_consensus.cc:348] T 16b46048dfd94f3cb5c7825a366687bb P 58e38d2ed0094a75a7b48af8719ba90e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } }
I20260925 04:54:36.742520 8630 raft_consensus.cc:374] T 16b46048dfd94f3cb5c7825a366687bb P 58e38d2ed0094a75a7b48af8719ba90e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:36.742532 8631 raft_consensus.cc:374] T 16b46048dfd94f3cb5c7825a366687bb P 0cb0df97fd14448d878f46990713c052 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:36.742576 8630 raft_consensus.cc:729] T 16b46048dfd94f3cb5c7825a366687bb P 58e38d2ed0094a75a7b48af8719ba90e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58e38d2ed0094a75a7b48af8719ba90e, State: Initialized, Role: FOLLOWER
I20260925 04:54:36.742619 8631 raft_consensus.cc:729] T 16b46048dfd94f3cb5c7825a366687bb P 0cb0df97fd14448d878f46990713c052 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb0df97fd14448d878f46990713c052, State: Initialized, Role: FOLLOWER
I20260925 04:54:36.742841 8630 consensus_queue.cc:260] T 16b46048dfd94f3cb5c7825a366687bb P 58e38d2ed0094a75a7b48af8719ba90e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } }
I20260925 04:54:36.742852 8631 consensus_queue.cc:260] T 16b46048dfd94f3cb5c7825a366687bb P 0cb0df97fd14448d878f46990713c052 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } }
I20260925 04:54:36.743551 8629 tablet_bootstrap.cc:492] T 16b46048dfd94f3cb5c7825a366687bb P 3ff76b5d25ad47a5897e7bce347ea4bc: No bootstrap required, opened a new log
I20260925 04:54:36.743691 8629 ts_tablet_manager.cc:1397] T 16b46048dfd94f3cb5c7825a366687bb P 3ff76b5d25ad47a5897e7bce347ea4bc: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20260925 04:54:36.744570 8629 raft_consensus.cc:348] T 16b46048dfd94f3cb5c7825a366687bb P 3ff76b5d25ad47a5897e7bce347ea4bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } }
I20260925 04:54:36.744808 8629 raft_consensus.cc:374] T 16b46048dfd94f3cb5c7825a366687bb P 3ff76b5d25ad47a5897e7bce347ea4bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:36.744933 8631 ts_tablet_manager.cc:1428] T 16b46048dfd94f3cb5c7825a366687bb P 0cb0df97fd14448d878f46990713c052: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 04:54:36.744910 8629 raft_consensus.cc:729] T 16b46048dfd94f3cb5c7825a366687bb P 3ff76b5d25ad47a5897e7bce347ea4bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ff76b5d25ad47a5897e7bce347ea4bc, State: Initialized, Role: FOLLOWER
I20260925 04:54:36.745237 8629 consensus_queue.cc:260] T 16b46048dfd94f3cb5c7825a366687bb P 3ff76b5d25ad47a5897e7bce347ea4bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } }
I20260925 04:54:36.745622 8630 ts_tablet_manager.cc:1428] T 16b46048dfd94f3cb5c7825a366687bb P 58e38d2ed0094a75a7b48af8719ba90e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260925 04:54:36.745939 8629 ts_tablet_manager.cc:1428] T 16b46048dfd94f3cb5c7825a366687bb P 3ff76b5d25ad47a5897e7bce347ea4bc: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260925 04:54:36.789724 8609 raft_consensus.cc:482] T 16b46048dfd94f3cb5c7825a366687bb P 3ff76b5d25ad47a5897e7bce347ea4bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:54:36.789970 8609 raft_consensus.cc:504] T 16b46048dfd94f3cb5c7825a366687bb P 3ff76b5d25ad47a5897e7bce347ea4bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } }
I20260925 04:54:36.790668 8609 leader_election.cc:288] T 16b46048dfd94f3cb5c7825a366687bb P 3ff76b5d25ad47a5897e7bce347ea4bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cb0df97fd14448d878f46990713c052 (127.6.29.129:45705), 58e38d2ed0094a75a7b48af8719ba90e (127.6.29.130:40465)
I20260925 04:54:36.791215 8319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16b46048dfd94f3cb5c7825a366687bb" candidate_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58e38d2ed0094a75a7b48af8719ba90e" is_pre_election: true
I20260925 04:54:36.791239 8180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16b46048dfd94f3cb5c7825a366687bb" candidate_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb0df97fd14448d878f46990713c052" is_pre_election: true
I20260925 04:54:36.791471 8180 raft_consensus.cc:2454] T 16b46048dfd94f3cb5c7825a366687bb P 0cb0df97fd14448d878f46990713c052 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ff76b5d25ad47a5897e7bce347ea4bc in term 0.
I20260925 04:54:36.791471 8319 raft_consensus.cc:2454] T 16b46048dfd94f3cb5c7825a366687bb P 58e38d2ed0094a75a7b48af8719ba90e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ff76b5d25ad47a5897e7bce347ea4bc in term 0.
I20260925 04:54:36.792016 8392 leader_election.cc:302] T 16b46048dfd94f3cb5c7825a366687bb P 3ff76b5d25ad47a5897e7bce347ea4bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cb0df97fd14448d878f46990713c052, 3ff76b5d25ad47a5897e7bce347ea4bc; no voters:
I20260925 04:54:36.792344 8609 raft_consensus.cc:2781] T 16b46048dfd94f3cb5c7825a366687bb P 3ff76b5d25ad47a5897e7bce347ea4bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:54:36.792485 8609 raft_consensus.cc:482] T 16b46048dfd94f3cb5c7825a366687bb P 3ff76b5d25ad47a5897e7bce347ea4bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:54:36.792582 8609 raft_consensus.cc:3037] T 16b46048dfd94f3cb5c7825a366687bb P 3ff76b5d25ad47a5897e7bce347ea4bc [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:36.794987 8609 raft_consensus.cc:504] T 16b46048dfd94f3cb5c7825a366687bb P 3ff76b5d25ad47a5897e7bce347ea4bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } }
I20260925 04:54:36.795567 8609 leader_election.cc:288] T 16b46048dfd94f3cb5c7825a366687bb P 3ff76b5d25ad47a5897e7bce347ea4bc [CANDIDATE]: Term 1 election: Requested vote from peers 0cb0df97fd14448d878f46990713c052 (127.6.29.129:45705), 58e38d2ed0094a75a7b48af8719ba90e (127.6.29.130:40465)
I20260925 04:54:36.796049 8180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16b46048dfd94f3cb5c7825a366687bb" candidate_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb0df97fd14448d878f46990713c052"
I20260925 04:54:36.796159 8319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16b46048dfd94f3cb5c7825a366687bb" candidate_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58e38d2ed0094a75a7b48af8719ba90e"
I20260925 04:54:36.796286 8180 raft_consensus.cc:3037] T 16b46048dfd94f3cb5c7825a366687bb P 0cb0df97fd14448d878f46990713c052 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:36.796399 8319 raft_consensus.cc:3037] T 16b46048dfd94f3cb5c7825a366687bb P 58e38d2ed0094a75a7b48af8719ba90e [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:36.799295 8180 raft_consensus.cc:2454] T 16b46048dfd94f3cb5c7825a366687bb P 0cb0df97fd14448d878f46990713c052 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ff76b5d25ad47a5897e7bce347ea4bc in term 1.
I20260925 04:54:36.799280 8319 raft_consensus.cc:2454] T 16b46048dfd94f3cb5c7825a366687bb P 58e38d2ed0094a75a7b48af8719ba90e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ff76b5d25ad47a5897e7bce347ea4bc in term 1.
I20260925 04:54:36.799733 8392 leader_election.cc:302] T 16b46048dfd94f3cb5c7825a366687bb P 3ff76b5d25ad47a5897e7bce347ea4bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cb0df97fd14448d878f46990713c052, 3ff76b5d25ad47a5897e7bce347ea4bc; no voters:
I20260925 04:54:36.800065 8609 raft_consensus.cc:2781] T 16b46048dfd94f3cb5c7825a366687bb P 3ff76b5d25ad47a5897e7bce347ea4bc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:54:36.800243 8609 raft_consensus.cc:686] T 16b46048dfd94f3cb5c7825a366687bb P 3ff76b5d25ad47a5897e7bce347ea4bc [term 1 LEADER]: Becoming Leader. State: Replica: 3ff76b5d25ad47a5897e7bce347ea4bc, State: Running, Role: LEADER
I20260925 04:54:36.800514 8609 consensus_queue.cc:237] T 16b46048dfd94f3cb5c7825a366687bb P 3ff76b5d25ad47a5897e7bce347ea4bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } }
I20260925 04:54:36.803356 8021 catalog_manager.cc:5495] T 16b46048dfd94f3cb5c7825a366687bb P 3ff76b5d25ad47a5897e7bce347ea4bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ff76b5d25ad47a5897e7bce347ea4bc (127.6.29.131). New cstate: current_term: 1 leader_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } health_report { overall_health: HEALTHY } } }
I20260925 04:54:36.827108 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:36.822127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39974 (local address 127.6.29.131:37063)
0925 04:54:36.822706 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:36.822710 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:36.822729 (+ 19us) server_negotiation.cc:407] Connection header received
0925 04:54:36.822779 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:36.822782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:36.822839 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:36.822921 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:54:36.823225 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:36.824107 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:36.825470 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:36.825928 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:36.826034 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:36.826381 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:54:36.826488 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:54:36.826553 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:36.826667 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 04:54:36.826755 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":298,"threads_started":1}
I20260925 04:54:36.836380 8180 raft_consensus.cc:1261] T 16b46048dfd94f3cb5c7825a366687bb P 0cb0df97fd14448d878f46990713c052 [term 1 FOLLOWER]: Refusing update from remote peer 3ff76b5d25ad47a5897e7bce347ea4bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:54:36.836397 8319 raft_consensus.cc:1261] T 16b46048dfd94f3cb5c7825a366687bb P 58e38d2ed0094a75a7b48af8719ba90e [term 1 FOLLOWER]: Refusing update from remote peer 3ff76b5d25ad47a5897e7bce347ea4bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:54:36.837010 8609 consensus_queue.cc:1035] T 16b46048dfd94f3cb5c7825a366687bb P 3ff76b5d25ad47a5897e7bce347ea4bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:54:36.837319 8612 consensus_queue.cc:1035] T 16b46048dfd94f3cb5c7825a366687bb P 3ff76b5d25ad47a5897e7bce347ea4bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:54:36.880019 8023 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:54:36.880242 8023 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:54:36.883042 8023 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3473cd541f7042a7903a56c2aa871845: Sending DeleteTablet for 3 replicas of tablet 16b46048dfd94f3cb5c7825a366687bb
I20260925 04:54:36.883867 8160 tablet_service.cc:1514] Processing DeleteTablet for tablet 16b46048dfd94f3cb5c7825a366687bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:54:36 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:59142
I20260925 04:54:36.884033 8438 tablet_service.cc:1514] Processing DeleteTablet for tablet 16b46048dfd94f3cb5c7825a366687bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:54:36 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:39966
I20260925 04:54:36.884161 8619 tablet_replica.cc:330] T 16b46048dfd94f3cb5c7825a366687bb P 0cb0df97fd14448d878f46990713c052: stopping tablet replica
I20260925 04:54:36.884312 8620 tablet_replica.cc:330] T 16b46048dfd94f3cb5c7825a366687bb P 3ff76b5d25ad47a5897e7bce347ea4bc: stopping tablet replica
I20260925 04:54:36.884354 8619 raft_consensus.cc:2229] T 16b46048dfd94f3cb5c7825a366687bb P 0cb0df97fd14448d878f46990713c052 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:54:36.884518 8620 raft_consensus.cc:2229] T 16b46048dfd94f3cb5c7825a366687bb P 3ff76b5d25ad47a5897e7bce347ea4bc [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:54:36.884574 8619 raft_consensus.cc:2258] T 16b46048dfd94f3cb5c7825a366687bb P 0cb0df97fd14448d878f46990713c052 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:54:36.884016 8299 tablet_service.cc:1514] Processing DeleteTablet for tablet 16b46048dfd94f3cb5c7825a366687bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:54:36 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:57260
I20260925 04:54:36.884883 8620 raft_consensus.cc:2258] T 16b46048dfd94f3cb5c7825a366687bb P 3ff76b5d25ad47a5897e7bce347ea4bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:54:36.885061 8618 tablet_replica.cc:330] T 16b46048dfd94f3cb5c7825a366687bb P 58e38d2ed0094a75a7b48af8719ba90e: stopping tablet replica
I20260925 04:54:36.885239 8618 raft_consensus.cc:2229] T 16b46048dfd94f3cb5c7825a366687bb P 58e38d2ed0094a75a7b48af8719ba90e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:54:36.885471 8618 raft_consensus.cc:2258] T 16b46048dfd94f3cb5c7825a366687bb P 58e38d2ed0094a75a7b48af8719ba90e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:54:36.885334 8023 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42608:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:54:36.885984 8023 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:54:36.886049 8619 ts_tablet_manager.cc:1905] T 16b46048dfd94f3cb5c7825a366687bb P 0cb0df97fd14448d878f46990713c052: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:54:36.886489 8618 ts_tablet_manager.cc:1905] T 16b46048dfd94f3cb5c7825a366687bb P 58e38d2ed0094a75a7b48af8719ba90e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:54:36.886825 8620 ts_tablet_manager.cc:1905] T 16b46048dfd94f3cb5c7825a366687bb P 3ff76b5d25ad47a5897e7bce347ea4bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:54:36.891496 8620 ts_tablet_manager.cc:1918] T 16b46048dfd94f3cb5c7825a366687bb P 3ff76b5d25ad47a5897e7bce347ea4bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:54:36.891610 8620 log.cc:1196] T 16b46048dfd94f3cb5c7825a366687bb P 3ff76b5d25ad47a5897e7bce347ea4bc: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-2/wal/wals/16b46048dfd94f3cb5c7825a366687bb
I20260925 04:54:36.892230 8620 ts_tablet_manager.cc:1939] T 16b46048dfd94f3cb5c7825a366687bb P 3ff76b5d25ad47a5897e7bce347ea4bc: Deleting consensus metadata
I20260925 04:54:36.892577 8619 ts_tablet_manager.cc:1918] T 16b46048dfd94f3cb5c7825a366687bb P 0cb0df97fd14448d878f46990713c052: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:54:36.892722 8619 log.cc:1196] T 16b46048dfd94f3cb5c7825a366687bb P 0cb0df97fd14448d878f46990713c052: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-0/wal/wals/16b46048dfd94f3cb5c7825a366687bb
I20260925 04:54:36.893121 8619 ts_tablet_manager.cc:1939] T 16b46048dfd94f3cb5c7825a366687bb P 0cb0df97fd14448d878f46990713c052: Deleting consensus metadata
I20260925 04:54:36.893693 8618 ts_tablet_manager.cc:1918] T 16b46048dfd94f3cb5c7825a366687bb P 58e38d2ed0094a75a7b48af8719ba90e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:54:36.893828 8618 log.cc:1196] T 16b46048dfd94f3cb5c7825a366687bb P 58e38d2ed0094a75a7b48af8719ba90e: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-1/wal/wals/16b46048dfd94f3cb5c7825a366687bb
I20260925 04:54:36.894094 8009 catalog_manager.cc:4841] TS 3ff76b5d25ad47a5897e7bce347ea4bc (127.6.29.131:37063): tablet 16b46048dfd94f3cb5c7825a366687bb (table test-table [id=bb3577e06aca4ad282d6b6db4f2eb885]) successfully deleted
I20260925 04:54:36.894184 8010 catalog_manager.cc:4841] TS 0cb0df97fd14448d878f46990713c052 (127.6.29.129:45705): tablet 16b46048dfd94f3cb5c7825a366687bb (table test-table [id=bb3577e06aca4ad282d6b6db4f2eb885]) successfully deleted
I20260925 04:54:36.894245 8618 ts_tablet_manager.cc:1939] T 16b46048dfd94f3cb5c7825a366687bb P 58e38d2ed0094a75a7b48af8719ba90e: Deleting consensus metadata
I20260925 04:54:36.895301 8007 catalog_manager.cc:4841] TS 58e38d2ed0094a75a7b48af8719ba90e (127.6.29.130:40465): tablet 16b46048dfd94f3cb5c7825a366687bb (table test-table [id=bb3577e06aca4ad282d6b6db4f2eb885]) successfully deleted
I20260925 04:54:36.897192 8160 tablet_service.cc:1467] Processing CreateTablet for tablet d6c379d859c340cba5bbd2482aa9f36d (DEFAULT_TABLE table=test-table [id=e1b0a5b8403248b0bea918e54c0f7980]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:54:36.897383 8299 tablet_service.cc:1467] Processing CreateTablet for tablet d6c379d859c340cba5bbd2482aa9f36d (DEFAULT_TABLE table=test-table [id=e1b0a5b8403248b0bea918e54c0f7980]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:54:36.897507 8438 tablet_service.cc:1467] Processing CreateTablet for tablet d6c379d859c340cba5bbd2482aa9f36d (DEFAULT_TABLE table=test-table [id=e1b0a5b8403248b0bea918e54c0f7980]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:54:36.897678 8160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6c379d859c340cba5bbd2482aa9f36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:54:36.897794 8299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6c379d859c340cba5bbd2482aa9f36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:54:36.897920 8438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6c379d859c340cba5bbd2482aa9f36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:54:36.902164 8631 tablet_bootstrap.cc:492] T d6c379d859c340cba5bbd2482aa9f36d P 0cb0df97fd14448d878f46990713c052: Bootstrap starting.
I20260925 04:54:36.903705 8631 tablet_bootstrap.cc:654] T d6c379d859c340cba5bbd2482aa9f36d P 0cb0df97fd14448d878f46990713c052: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:36.904626 8630 tablet_bootstrap.cc:492] T d6c379d859c340cba5bbd2482aa9f36d P 58e38d2ed0094a75a7b48af8719ba90e: Bootstrap starting.
I20260925 04:54:36.905017 8629 tablet_bootstrap.cc:492] T d6c379d859c340cba5bbd2482aa9f36d P 3ff76b5d25ad47a5897e7bce347ea4bc: Bootstrap starting.
I20260925 04:54:36.905557 8631 tablet_bootstrap.cc:492] T d6c379d859c340cba5bbd2482aa9f36d P 0cb0df97fd14448d878f46990713c052: No bootstrap required, opened a new log
I20260925 04:54:36.905776 8631 ts_tablet_manager.cc:1397] T d6c379d859c340cba5bbd2482aa9f36d P 0cb0df97fd14448d878f46990713c052: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 04:54:36.906364 8630 tablet_bootstrap.cc:654] T d6c379d859c340cba5bbd2482aa9f36d P 58e38d2ed0094a75a7b48af8719ba90e: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:36.906419 8631 raft_consensus.cc:348] T d6c379d859c340cba5bbd2482aa9f36d P 0cb0df97fd14448d878f46990713c052 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } }
I20260925 04:54:36.906631 8629 tablet_bootstrap.cc:654] T d6c379d859c340cba5bbd2482aa9f36d P 3ff76b5d25ad47a5897e7bce347ea4bc: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:36.906661 8631 raft_consensus.cc:374] T d6c379d859c340cba5bbd2482aa9f36d P 0cb0df97fd14448d878f46990713c052 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:36.906746 8631 raft_consensus.cc:729] T d6c379d859c340cba5bbd2482aa9f36d P 0cb0df97fd14448d878f46990713c052 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cb0df97fd14448d878f46990713c052, State: Initialized, Role: FOLLOWER
I20260925 04:54:36.907011 8631 consensus_queue.cc:260] T d6c379d859c340cba5bbd2482aa9f36d P 0cb0df97fd14448d878f46990713c052 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } }
I20260925 04:54:36.908563 8631 ts_tablet_manager.cc:1428] T d6c379d859c340cba5bbd2482aa9f36d P 0cb0df97fd14448d878f46990713c052: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 04:54:36.909345 8630 tablet_bootstrap.cc:492] T d6c379d859c340cba5bbd2482aa9f36d P 58e38d2ed0094a75a7b48af8719ba90e: No bootstrap required, opened a new log
I20260925 04:54:36.909516 8630 ts_tablet_manager.cc:1397] T d6c379d859c340cba5bbd2482aa9f36d P 58e38d2ed0094a75a7b48af8719ba90e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 04:54:36.910256 8630 raft_consensus.cc:348] T d6c379d859c340cba5bbd2482aa9f36d P 58e38d2ed0094a75a7b48af8719ba90e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } }
I20260925 04:54:36.910533 8630 raft_consensus.cc:374] T d6c379d859c340cba5bbd2482aa9f36d P 58e38d2ed0094a75a7b48af8719ba90e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:36.910614 8630 raft_consensus.cc:729] T d6c379d859c340cba5bbd2482aa9f36d P 58e38d2ed0094a75a7b48af8719ba90e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58e38d2ed0094a75a7b48af8719ba90e, State: Initialized, Role: FOLLOWER
I20260925 04:54:36.910856 8630 consensus_queue.cc:260] T d6c379d859c340cba5bbd2482aa9f36d P 58e38d2ed0094a75a7b48af8719ba90e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } }
I20260925 04:54:36.911293 8630 ts_tablet_manager.cc:1428] T d6c379d859c340cba5bbd2482aa9f36d P 58e38d2ed0094a75a7b48af8719ba90e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 04:54:36.912516 8629 tablet_bootstrap.cc:492] T d6c379d859c340cba5bbd2482aa9f36d P 3ff76b5d25ad47a5897e7bce347ea4bc: No bootstrap required, opened a new log
I20260925 04:54:36.912700 8629 ts_tablet_manager.cc:1397] T d6c379d859c340cba5bbd2482aa9f36d P 3ff76b5d25ad47a5897e7bce347ea4bc: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20260925 04:54:36.913501 8629 raft_consensus.cc:348] T d6c379d859c340cba5bbd2482aa9f36d P 3ff76b5d25ad47a5897e7bce347ea4bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } }
I20260925 04:54:36.913692 8629 raft_consensus.cc:374] T d6c379d859c340cba5bbd2482aa9f36d P 3ff76b5d25ad47a5897e7bce347ea4bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:36.913740 8629 raft_consensus.cc:729] T d6c379d859c340cba5bbd2482aa9f36d P 3ff76b5d25ad47a5897e7bce347ea4bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ff76b5d25ad47a5897e7bce347ea4bc, State: Initialized, Role: FOLLOWER
I20260925 04:54:36.913998 8629 consensus_queue.cc:260] T d6c379d859c340cba5bbd2482aa9f36d P 3ff76b5d25ad47a5897e7bce347ea4bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } }
I20260925 04:54:36.914515 8629 ts_tablet_manager.cc:1428] T d6c379d859c340cba5bbd2482aa9f36d P 3ff76b5d25ad47a5897e7bce347ea4bc: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 04:54:37.101804 8596 raft_consensus.cc:482] T d6c379d859c340cba5bbd2482aa9f36d P 3ff76b5d25ad47a5897e7bce347ea4bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:54:37.102108 8596 raft_consensus.cc:504] T d6c379d859c340cba5bbd2482aa9f36d P 3ff76b5d25ad47a5897e7bce347ea4bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } }
I20260925 04:54:37.102952 8596 leader_election.cc:288] T d6c379d859c340cba5bbd2482aa9f36d P 3ff76b5d25ad47a5897e7bce347ea4bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cb0df97fd14448d878f46990713c052 (127.6.29.129:45705), 58e38d2ed0094a75a7b48af8719ba90e (127.6.29.130:40465)
I20260925 04:54:37.103544 8179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c379d859c340cba5bbd2482aa9f36d" candidate_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb0df97fd14448d878f46990713c052" is_pre_election: true
I20260925 04:54:37.103546 8319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c379d859c340cba5bbd2482aa9f36d" candidate_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58e38d2ed0094a75a7b48af8719ba90e" is_pre_election: true
I20260925 04:54:37.103845 8319 raft_consensus.cc:2454] T d6c379d859c340cba5bbd2482aa9f36d P 58e38d2ed0094a75a7b48af8719ba90e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ff76b5d25ad47a5897e7bce347ea4bc in term 0.
I20260925 04:54:37.103843 8179 raft_consensus.cc:2454] T d6c379d859c340cba5bbd2482aa9f36d P 0cb0df97fd14448d878f46990713c052 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ff76b5d25ad47a5897e7bce347ea4bc in term 0.
I20260925 04:54:37.104302 8389 leader_election.cc:302] T d6c379d859c340cba5bbd2482aa9f36d P 3ff76b5d25ad47a5897e7bce347ea4bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ff76b5d25ad47a5897e7bce347ea4bc, 58e38d2ed0094a75a7b48af8719ba90e; no voters:
I20260925 04:54:37.104593 8596 raft_consensus.cc:2781] T d6c379d859c340cba5bbd2482aa9f36d P 3ff76b5d25ad47a5897e7bce347ea4bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:54:37.104717 8596 raft_consensus.cc:482] T d6c379d859c340cba5bbd2482aa9f36d P 3ff76b5d25ad47a5897e7bce347ea4bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:54:37.104818 8596 raft_consensus.cc:3037] T d6c379d859c340cba5bbd2482aa9f36d P 3ff76b5d25ad47a5897e7bce347ea4bc [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:37.106822 8596 raft_consensus.cc:504] T d6c379d859c340cba5bbd2482aa9f36d P 3ff76b5d25ad47a5897e7bce347ea4bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } }
I20260925 04:54:37.107456 8596 leader_election.cc:288] T d6c379d859c340cba5bbd2482aa9f36d P 3ff76b5d25ad47a5897e7bce347ea4bc [CANDIDATE]: Term 1 election: Requested vote from peers 0cb0df97fd14448d878f46990713c052 (127.6.29.129:45705), 58e38d2ed0094a75a7b48af8719ba90e (127.6.29.130:40465)
I20260925 04:54:37.107946 8179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c379d859c340cba5bbd2482aa9f36d" candidate_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cb0df97fd14448d878f46990713c052"
I20260925 04:54:37.108017 8319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6c379d859c340cba5bbd2482aa9f36d" candidate_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58e38d2ed0094a75a7b48af8719ba90e"
I20260925 04:54:37.108137 8179 raft_consensus.cc:3037] T d6c379d859c340cba5bbd2482aa9f36d P 0cb0df97fd14448d878f46990713c052 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:37.108273 8319 raft_consensus.cc:3037] T d6c379d859c340cba5bbd2482aa9f36d P 58e38d2ed0094a75a7b48af8719ba90e [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:37.110224 8179 raft_consensus.cc:2454] T d6c379d859c340cba5bbd2482aa9f36d P 0cb0df97fd14448d878f46990713c052 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ff76b5d25ad47a5897e7bce347ea4bc in term 1.
I20260925 04:54:37.110231 8319 raft_consensus.cc:2454] T d6c379d859c340cba5bbd2482aa9f36d P 58e38d2ed0094a75a7b48af8719ba90e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ff76b5d25ad47a5897e7bce347ea4bc in term 1.
I20260925 04:54:37.110668 8392 leader_election.cc:302] T d6c379d859c340cba5bbd2482aa9f36d P 3ff76b5d25ad47a5897e7bce347ea4bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cb0df97fd14448d878f46990713c052, 3ff76b5d25ad47a5897e7bce347ea4bc; no voters:
I20260925 04:54:37.110973 8596 raft_consensus.cc:2781] T d6c379d859c340cba5bbd2482aa9f36d P 3ff76b5d25ad47a5897e7bce347ea4bc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:54:37.111147 8596 raft_consensus.cc:686] T d6c379d859c340cba5bbd2482aa9f36d P 3ff76b5d25ad47a5897e7bce347ea4bc [term 1 LEADER]: Becoming Leader. State: Replica: 3ff76b5d25ad47a5897e7bce347ea4bc, State: Running, Role: LEADER
I20260925 04:54:37.111414 8596 consensus_queue.cc:237] T d6c379d859c340cba5bbd2482aa9f36d P 3ff76b5d25ad47a5897e7bce347ea4bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } }
I20260925 04:54:37.113955 8023 catalog_manager.cc:5495] T d6c379d859c340cba5bbd2482aa9f36d P 3ff76b5d25ad47a5897e7bce347ea4bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ff76b5d25ad47a5897e7bce347ea4bc (127.6.29.131). New cstate: current_term: 1 leader_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ff76b5d25ad47a5897e7bce347ea4bc" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 37063 } health_report { overall_health: HEALTHY } } }
I20260925 04:54:37.164184 8179 raft_consensus.cc:1261] T d6c379d859c340cba5bbd2482aa9f36d P 0cb0df97fd14448d878f46990713c052 [term 1 FOLLOWER]: Refusing update from remote peer 3ff76b5d25ad47a5897e7bce347ea4bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:54:37.164191 8319 raft_consensus.cc:1261] T d6c379d859c340cba5bbd2482aa9f36d P 58e38d2ed0094a75a7b48af8719ba90e [term 1 FOLLOWER]: Refusing update from remote peer 3ff76b5d25ad47a5897e7bce347ea4bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:54:37.164731 8612 consensus_queue.cc:1035] T d6c379d859c340cba5bbd2482aa9f36d P 3ff76b5d25ad47a5897e7bce347ea4bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cb0df97fd14448d878f46990713c052" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 45705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:54:37.165007 8596 consensus_queue.cc:1035] T d6c379d859c340cba5bbd2482aa9f36d P 3ff76b5d25ad47a5897e7bce347ea4bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "58e38d2ed0094a75a7b48af8719ba90e" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 04:54:37.182018 8600 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 04:54:38.766360 8021 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:54:38.766618 8021 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:54:38.770553 8021 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3473cd541f7042a7903a56c2aa871845: Sending DeleteTablet for 3 replicas of tablet d6c379d859c340cba5bbd2482aa9f36d
I20260925 04:54:38.771409 8299 tablet_service.cc:1514] Processing DeleteTablet for tablet d6c379d859c340cba5bbd2482aa9f36d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:54:38 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:57260
I20260925 04:54:38.772125 8674 tablet_replica.cc:330] T d6c379d859c340cba5bbd2482aa9f36d P 58e38d2ed0094a75a7b48af8719ba90e: stopping tablet replica
I20260925 04:54:38.772171 8160 tablet_service.cc:1514] Processing DeleteTablet for tablet d6c379d859c340cba5bbd2482aa9f36d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:54:38 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:59142
I20260925 04:54:38.772413 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 8094
I20260925 04:54:38.772485 8674 raft_consensus.cc:2229] T d6c379d859c340cba5bbd2482aa9f36d P 58e38d2ed0094a75a7b48af8719ba90e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:54:38.772517 8438 tablet_service.cc:1514] Processing DeleteTablet for tablet d6c379d859c340cba5bbd2482aa9f36d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:54:38 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:39966
I20260925 04:54:38.772730 8674 raft_consensus.cc:2258] T d6c379d859c340cba5bbd2482aa9f36d P 58e38d2ed0094a75a7b48af8719ba90e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:54:38.779032 8674 ts_tablet_manager.cc:1905] T d6c379d859c340cba5bbd2482aa9f36d P 58e38d2ed0094a75a7b48af8719ba90e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:54:38.784997 8676 tablet_replica.cc:330] T d6c379d859c340cba5bbd2482aa9f36d P 3ff76b5d25ad47a5897e7bce347ea4bc: stopping tablet replica
I20260925 04:54:38.785703 8675 tablet_replica.cc:330] T d6c379d859c340cba5bbd2482aa9f36d P 0cb0df97fd14448d878f46990713c052: stopping tablet replica
I20260925 04:54:38.785907 8675 raft_consensus.cc:2229] T d6c379d859c340cba5bbd2482aa9f36d P 0cb0df97fd14448d878f46990713c052 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:54:38.786134 8675 raft_consensus.cc:2258] T d6c379d859c340cba5bbd2482aa9f36d P 0cb0df97fd14448d878f46990713c052 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:54:38.786700 8676 raft_consensus.cc:2229] T d6c379d859c340cba5bbd2482aa9f36d P 3ff76b5d25ad47a5897e7bce347ea4bc [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:54:38.787089 8676 raft_consensus.cc:2258] T d6c379d859c340cba5bbd2482aa9f36d P 3ff76b5d25ad47a5897e7bce347ea4bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:54:38.787369 8674 ts_tablet_manager.cc:1918] T d6c379d859c340cba5bbd2482aa9f36d P 58e38d2ed0094a75a7b48af8719ba90e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 04:54:38.787503 8674 log.cc:1196] T d6c379d859c340cba5bbd2482aa9f36d P 58e38d2ed0094a75a7b48af8719ba90e: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-1/wal/wals/d6c379d859c340cba5bbd2482aa9f36d
I20260925 04:54:38.787972 8674 ts_tablet_manager.cc:1939] T d6c379d859c340cba5bbd2482aa9f36d P 58e38d2ed0094a75a7b48af8719ba90e: Deleting consensus metadata
I20260925 04:54:38.787968 8675 ts_tablet_manager.cc:1905] T d6c379d859c340cba5bbd2482aa9f36d P 0cb0df97fd14448d878f46990713c052: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:54:38.789103 8007 catalog_manager.cc:4841] TS 58e38d2ed0094a75a7b48af8719ba90e (127.6.29.130:40465): tablet d6c379d859c340cba5bbd2482aa9f36d (table test-table [id=e1b0a5b8403248b0bea918e54c0f7980]) successfully deleted
I20260925 04:54:38.792452 8676 ts_tablet_manager.cc:1905] T d6c379d859c340cba5bbd2482aa9f36d P 3ff76b5d25ad47a5897e7bce347ea4bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:54:38.793473 8675 ts_tablet_manager.cc:1918] T d6c379d859c340cba5bbd2482aa9f36d P 0cb0df97fd14448d878f46990713c052: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 04:54:38.793604 8675 log.cc:1196] T d6c379d859c340cba5bbd2482aa9f36d P 0cb0df97fd14448d878f46990713c052: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-0/wal/wals/d6c379d859c340cba5bbd2482aa9f36d
I20260925 04:54:38.794070 8675 ts_tablet_manager.cc:1939] T d6c379d859c340cba5bbd2482aa9f36d P 0cb0df97fd14448d878f46990713c052: Deleting consensus metadata
Sep 25 04:54:38 dist-test-slave-xzjr krb5kdc[7978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312076, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.29.129@KRBTEST.COM
I20260925 04:54:38.795173 8010 catalog_manager.cc:4841] TS 0cb0df97fd14448d878f46990713c052 (127.6.29.129:45705): tablet d6c379d859c340cba5bbd2482aa9f36d (table test-table [id=e1b0a5b8403248b0bea918e54c0f7980]) successfully deleted
I20260925 04:54:38.798070 8676 ts_tablet_manager.cc:1918] T d6c379d859c340cba5bbd2482aa9f36d P 3ff76b5d25ad47a5897e7bce347ea4bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 04:54:38.798221 8676 log.cc:1196] T d6c379d859c340cba5bbd2482aa9f36d P 3ff76b5d25ad47a5897e7bce347ea4bc: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790312053666503-6262-0/minicluster-data/ts-2/wal/wals/d6c379d859c340cba5bbd2482aa9f36d
I20260925 04:54:38.798610 8676 ts_tablet_manager.cc:1939] T d6c379d859c340cba5bbd2482aa9f36d P 3ff76b5d25ad47a5897e7bce347ea4bc: Deleting consensus metadata
I20260925 04:54:38.799533 8009 catalog_manager.cc:4841] TS 3ff76b5d25ad47a5897e7bce347ea4bc (127.6.29.131:37063): tablet d6c379d859c340cba5bbd2482aa9f36d (table test-table [id=e1b0a5b8403248b0bea918e54c0f7980]) successfully deleted
I20260925 04:54:38.801931 8678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:38.773906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59160 (local address 127.6.29.129:45705)
0925 04:54:38.785818 (+ 11912us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:38.785826 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:38.786416 (+ 590us) server_negotiation.cc:407] Connection header received
0925 04:54:38.786717 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:38.786722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:38.786833 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:38.786942 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:38.788679 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:38.789722 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:38.790963 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:38.791406 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:38.791582 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:38.795953 (+ 4371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:38.795968 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:38.795986 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:38.796018 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:38.799048 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:38.799856 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:38.799862 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:38.799865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:38.799989 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:38.800391 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:38.800395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:38.800397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:38.800725 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:38.801041 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:38.801340 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 04:54:38.801571 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11753,"thread_start_us":11626,"threads_started":1}
W20260925 04:54:38.802546 8225 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59160
W20260925 04:54:38.803267 6262 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 8094: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 04:54:38.803481 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 8094
I20260925 04:54:38.825750 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 8234
Sep 25 04:54:38 dist-test-slave-xzjr krb5kdc[7978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312076, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.29.130@KRBTEST.COM
I20260925 04:54:38.841807 8679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:38.826539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57278 (local address 127.6.29.130:40465)
0925 04:54:38.827108 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:38.827112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:38.827127 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:54:38.827197 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:38.827201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:38.827263 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:38.827386 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:38.828696 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:38.829411 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:38.830670 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:38.831202 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:38.831347 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:38.835561 (+ 4214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:38.835575 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:38.835592 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:38.835625 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:38.838944 (+ 3319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:38.839652 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:38.839660 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:38.839662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:38.839854 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:38.840222 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:38.840228 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:38.840232 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:38.840581 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:38.840896 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:38.841213 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 04:54:38.841448 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":300,"threads_started":1}
W20260925 04:54:38.842551 8364 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57278
W20260925 04:54:38.843185 6262 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 8234: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 04:54:38.843365 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 8234
I20260925 04:54:38.865586 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 8373
Sep 25 04:54:38 dist-test-slave-xzjr krb5kdc[7978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312076, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.29.131@KRBTEST.COM
I20260925 04:54:38.880928 8680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:38.866392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39978 (local address 127.6.29.131:37063)
0925 04:54:38.866912 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:38.866916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:38.866931 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:54:38.866984 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:38.866987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:38.867049 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:38.867137 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:38.868511 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:38.869395 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:38.870527 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:38.870922 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:38.871011 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:38.875265 (+ 4254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:38.875279 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:38.875296 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:38.875351 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:38.878209 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:38.878971 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:38.878979 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:38.878981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:38.879120 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:38.879442 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:38.879445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:38.879447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:38.879907 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:38.880140 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:38.880373 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 04:54:38.880538 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":296,"threads_started":1}
W20260925 04:54:38.881610 8503 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39978
W20260925 04:54:38.882268 6262 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 8373: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 04:54:38.882418 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 8373
I20260925 04:54:38.904408 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 7991
I20260925 04:54:38.905632 8063 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:54:39.020323 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 7991
2026-09-25T04:54:39Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8418 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:54:39 dist-test-slave-xzjr krb5kdc[8684](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:54:39 dist-test-slave-xzjr krb5kdc[8684](info): set up 2 sockets
Sep 25 04:54:39 dist-test-slave-xzjr krb5kdc[8684](info): commencing operation
krb5kdc: starting...
W20260925 04:54:40.116670 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.005s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 04:54:40 dist-test-slave-xzjr krb5kdc[8684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:54:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:54:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.29.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
I20260925 04:54:40.317034 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:44813
--webserver_interface=127.6.29.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:33995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:44813
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:54:40.641111 8697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:54:40.641423 8697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:54:40.641580 8697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:54:40.648934 8697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:54:40.649047 8697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:54:40.649135 8697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:54:40.649242 8697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:54:40.649307 8697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:54:40.659257 8697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:33995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:44813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:44813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:54:40.660691 8697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:54:40.662012 8697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:54:40.674288 8705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:40.674320 8703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:40.674259 8702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:54:40.674927 8697 server_base.cc:1034] running on GCE node
I20260925 04:54:40.675890 8697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:54:40.677310 8697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:54:40.678563 8697 hybrid_clock.cc:648] HybridClock initialized: now 1790312080678516 us; error 51 us; skew 500 ppm
I20260925 04:54:40.678941 8697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:54:40 dist-test-slave-xzjr krb5kdc[8684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312080, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:54:40.682024 8697 init.cc:377] Logged in from keytab as kudu/127.6.29.190@KRBTEST.COM (short username kudu)
I20260925 04:54:40.683478 8697 webserver.cc:460] Webserver started at http://127.6.29.190:45787/ using document root <none> and password file <none>
I20260925 04:54:40.684242 8697 fs_manager.cc:359] Metadata directory not provided
I20260925 04:54:40.684347 8697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:54:40.684592 8697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:54:40.687484 8697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "8a2947bc136f4710a4278adcc5ecbe8a"
format_stamp: "Formatted at 2026-09-25 04:54:40 on dist-test-slave-xzjr"
server_key: "bae1598a82d5575927c36a62be03b0f8"
server_key_iv: "7ff859ba6110f2c1ab388dcfbbd67cd9"
server_key_version: "encryptionkey@0"
I20260925 04:54:40.688313 8697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "8a2947bc136f4710a4278adcc5ecbe8a"
format_stamp: "Formatted at 2026-09-25 04:54:40 on dist-test-slave-xzjr"
server_key: "bae1598a82d5575927c36a62be03b0f8"
server_key_iv: "7ff859ba6110f2c1ab388dcfbbd67cd9"
server_key_version: "encryptionkey@0"
I20260925 04:54:40.693652 8697 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 04:54:40.697146 8712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:40.698526 8697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:54:40.698683 8697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "8a2947bc136f4710a4278adcc5ecbe8a"
format_stamp: "Formatted at 2026-09-25 04:54:40 on dist-test-slave-xzjr"
server_key: "bae1598a82d5575927c36a62be03b0f8"
server_key_iv: "7ff859ba6110f2c1ab388dcfbbd67cd9"
server_key_version: "encryptionkey@0"
I20260925 04:54:40.698864 8697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:54:40.713019 8697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:54:40.717059 8697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:54:40.717294 8697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:54:40.735659 8697 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:44813
I20260925 04:54:40.735680 8763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:44813 every 8 connection(s)
I20260925 04:54:40.737103 8697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:54:40.740916 8764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:54:40.745623 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 8697
I20260925 04:54:40.745893 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 04:54:40.746562 6262 external_mini_cluster.cc:1428] Setting key 90cb73a0a8ff7d730de9404894299ad2
I20260925 04:54:40.752699 8764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a2947bc136f4710a4278adcc5ecbe8a: Bootstrap starting.
I20260925 04:54:40.756616 8764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a2947bc136f4710a4278adcc5ecbe8a: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:40.757644 8764 log.cc:824] T 00000000000000000000000000000000 P 8a2947bc136f4710a4278adcc5ecbe8a: Log is configured to *not* fsync() on all Append() calls
I20260925 04:54:40.760682 8764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a2947bc136f4710a4278adcc5ecbe8a: No bootstrap required, opened a new log
Sep 25 04:54:40 dist-test-slave-xzjr krb5kdc[8684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:54:40.771101 8764 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a2947bc136f4710a4278adcc5ecbe8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a2947bc136f4710a4278adcc5ecbe8a" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 44813 } }
I20260925 04:54:40.771534 8764 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a2947bc136f4710a4278adcc5ecbe8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:40.771646 8764 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a2947bc136f4710a4278adcc5ecbe8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2947bc136f4710a4278adcc5ecbe8a, State: Initialized, Role: FOLLOWER
I20260925 04:54:40.771920 8767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:40.748763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42114 (local address 127.6.29.190:44813)
0925 04:54:40.749690 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:40.749711 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:40.749761 (+ 50us) server_negotiation.cc:407] Connection header received
0925 04:54:40.752122 (+ 2361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:40.752149 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:40.752657 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:40.753099 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:40.754434 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:40.756561 (+ 2127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:40.758004 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:40.758894 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:40.759074 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:40.763108 (+ 4034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:40.763120 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:40.763139 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:40.763195 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:40.767367 (+ 4172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:40.768162 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:40.768171 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:40.768179 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:40.768369 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:40.768714 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:40.768719 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:40.768722 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:40.769450 (+ 728us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:40.769735 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:40.770138 (+ 403us) server_negotiation.cc:299] Negotiation successful
0925 04:54:40.770548 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":417,"threads_started":1}
I20260925 04:54:40.772364 8764 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a2947bc136f4710a4278adcc5ecbe8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a2947bc136f4710a4278adcc5ecbe8a" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 44813 } }
I20260925 04:54:40.772615 8764 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a2947bc136f4710a4278adcc5ecbe8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:54:40.772744 8764 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a2947bc136f4710a4278adcc5ecbe8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:54:40.772939 8764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a2947bc136f4710a4278adcc5ecbe8a [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:40.775193 8764 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a2947bc136f4710a4278adcc5ecbe8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a2947bc136f4710a4278adcc5ecbe8a" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 44813 } }
I20260925 04:54:40.775694 8764 leader_election.cc:302] T 00000000000000000000000000000000 P 8a2947bc136f4710a4278adcc5ecbe8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a2947bc136f4710a4278adcc5ecbe8a; no voters:
I20260925 04:54:40.776347 8764 leader_election.cc:288] T 00000000000000000000000000000000 P 8a2947bc136f4710a4278adcc5ecbe8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:54:40.776489 8769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a2947bc136f4710a4278adcc5ecbe8a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:54:40.777212 8769 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a2947bc136f4710a4278adcc5ecbe8a [term 1 LEADER]: Becoming Leader. State: Replica: 8a2947bc136f4710a4278adcc5ecbe8a, State: Running, Role: LEADER
I20260925 04:54:40.777799 8769 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a2947bc136f4710a4278adcc5ecbe8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a2947bc136f4710a4278adcc5ecbe8a" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 44813 } }
I20260925 04:54:40.778581 8764 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a2947bc136f4710a4278adcc5ecbe8a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:54:40.780824 8770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a2947bc136f4710a4278adcc5ecbe8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a2947bc136f4710a4278adcc5ecbe8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a2947bc136f4710a4278adcc5ecbe8a" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 44813 } } }
I20260925 04:54:40.781332 8770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a2947bc136f4710a4278adcc5ecbe8a [sys.catalog]: This master's current role is: LEADER
I20260925 04:54:40.781278 8771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a2947bc136f4710a4278adcc5ecbe8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a2947bc136f4710a4278adcc5ecbe8a. Latest consensus state: current_term: 1 leader_uuid: "8a2947bc136f4710a4278adcc5ecbe8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a2947bc136f4710a4278adcc5ecbe8a" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 44813 } } }
I20260925 04:54:40.781562 8771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a2947bc136f4710a4278adcc5ecbe8a [sys.catalog]: This master's current role is: LEADER
I20260925 04:54:40.785902 8778 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:54:40.790323 8778 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:54:40.798264 8778 catalog_manager.cc:1347] Generated new cluster ID: ef3700747a774a8c9ebd2bc2ef167a65
I20260925 04:54:40.798388 8778 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:54:40.839334 8778 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:54:40.841353 8778 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:54:40.848928 8778 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a2947bc136f4710a4278adcc5ecbe8a: Generated new TSK 0
I20260925 04:54:40.850051 8778 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.29.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab.
Entry for principal kudu/127.6.29.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab.
I20260925 04:54:40.904461 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.129:0
--local_ip_for_outbound_sockets=127.6.29.129
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:44813
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:33995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:54:41.232834 8790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:54:41.233121 8790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:54:41.233214 8790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:54:41.240496 8790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:54:41.240617 8790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:54:41.240734 8790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.129
I20260925 04:54:41.250732 8790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:33995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:44813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.129
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:54:41.252091 8790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:54:41.253376 8790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:54:41.266660 8798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:41.266724 8795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:41.266774 8796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:54:41.269502 8790 server_base.cc:1034] running on GCE node
I20260925 04:54:41.270156 8790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:54:41.271159 8790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:54:41.272367 8790 hybrid_clock.cc:648] HybridClock initialized: now 1790312081272326 us; error 42 us; skew 500 ppm
I20260925 04:54:41.272697 8790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:54:41 dist-test-slave-xzjr krb5kdc[8684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312081, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:54:41.282871 8790 init.cc:377] Logged in from keytab as kudu/127.6.29.129@KRBTEST.COM (short username kudu)
I20260925 04:54:41.284595 8790 webserver.cc:460] Webserver started at http://127.6.29.129:33183/ using document root <none> and password file <none>
I20260925 04:54:41.285390 8790 fs_manager.cc:359] Metadata directory not provided
I20260925 04:54:41.285538 8790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:54:41.285864 8790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:54:41.288847 8790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-0/data/instance:
uuid: "00292e484b814430a2e675ba60b4221d"
format_stamp: "Formatted at 2026-09-25 04:54:41 on dist-test-slave-xzjr"
server_key: "371d65f4d87aa3d17ad96bfda2f0d198"
server_key_iv: "4aa45caa9ffcc84dd2681974b7847f1c"
server_key_version: "encryptionkey@0"
I20260925 04:54:41.289608 8790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance:
uuid: "00292e484b814430a2e675ba60b4221d"
format_stamp: "Formatted at 2026-09-25 04:54:41 on dist-test-slave-xzjr"
server_key: "371d65f4d87aa3d17ad96bfda2f0d198"
server_key_iv: "4aa45caa9ffcc84dd2681974b7847f1c"
server_key_version: "encryptionkey@0"
I20260925 04:54:41.294783 8790 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 04:54:41.298471 8805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:41.299925 8790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:54:41.300107 8790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-0/wal
uuid: "00292e484b814430a2e675ba60b4221d"
format_stamp: "Formatted at 2026-09-25 04:54:41 on dist-test-slave-xzjr"
server_key: "371d65f4d87aa3d17ad96bfda2f0d198"
server_key_iv: "4aa45caa9ffcc84dd2681974b7847f1c"
server_key_version: "encryptionkey@0"
I20260925 04:54:41.300323 8790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:54:41.314174 8790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:54:41.318207 8790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:54:41.318496 8790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:54:41.319427 8790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:54:41.320822 8790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:54:41.320925 8790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:41.321079 8790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:54:41.321159 8790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:41.353945 8790 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.129:42063
I20260925 04:54:41.353991 8917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.129:42063 every 8 connection(s)
I20260925 04:54:41.355229 8790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
I20260925 04:54:41.365062 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 8790
I20260925 04:54:41.365269 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance
I20260925 04:54:41.366052 6262 external_mini_cluster.cc:1428] Setting key 1d374fdef25089fb50f341d788dafbb2
Sep 25 04:54:41 dist-test-slave-xzjr krb5kdc[8684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312081, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:54:41.375651 8920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:41.357942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:45639 (local address 127.6.29.190:44813)
0925 04:54:41.358398 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:41.358411 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:41.359399 (+ 988us) server_negotiation.cc:407] Connection header received
0925 04:54:41.361233 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:41.361237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:41.361303 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:41.361407 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:41.362877 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:41.363600 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:41.365709 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:41.366154 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:41.366276 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:41.370073 (+ 3797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:41.370077 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:41.370081 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:41.370111 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:41.372720 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:41.373429 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:41.373442 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:41.373444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:41.373552 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:41.373975 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:41.373980 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:41.373984 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:41.374227 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:41.374379 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:41.375067 (+ 688us) server_negotiation.cc:299] Negotiation successful
0925 04:54:41.375283 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":231,"threads_started":1}
I20260925 04:54:41.376631 8921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:41.358516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:44813 (local address 127.6.29.129:45639)
0925 04:54:41.359229 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:41.359272 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:41.360960 (+ 1688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:41.361620 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:41.361629 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:41.362138 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:54:41.362716 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:41.362732 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:41.363724 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:41.363727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:41.365547 (+ 1820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:41.365563 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:41.366392 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:41.366401 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:41.366684 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:41.367404 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:54:41.367427 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:54:41.369848 (+ 2421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:54:41.372876 (+ 3028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:41.372886 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:41.372903 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:41.373305 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:41.373660 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:41.373665 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:41.373668 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:41.373858 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:41.374359 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:54:41.374367 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:54:41.374703 (+ 336us) client_negotiation.cc:769] Sending connection context
0925 04:54:41.374964 (+ 261us) client_negotiation.cc:240] Negotiation successful
0925 04:54:41.375492 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":269,"threads_started":1}
I20260925 04:54:41.378226 8918 heartbeater.cc:347] Connected to a master server at 127.6.29.190:44813
I20260925 04:54:41.378569 8918 heartbeater.cc:464] Registering TS with master...
I20260925 04:54:41.379380 8918 heartbeater.cc:511] Master 127.6.29.190:44813 requested a full tablet report, sending...
I20260925 04:54:41.381877 8729 ts_manager.cc:194] Registered new tserver with Master: 00292e484b814430a2e675ba60b4221d (127.6.29.129:42063)
WARNING: no policy specified for kudu/127.6.29.130@KRBTEST.COM; defaulting to no policy
I20260925 04:54:41.384083 8729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.129@KRBTEST.COM'} at 127.6.29.129:45639
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab.
Entry for principal kudu/127.6.29.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab.
I20260925 04:54:41.411595 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.130:0
--local_ip_for_outbound_sockets=127.6.29.130
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:44813
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:33995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:54:41.734771 8924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:54:41.735169 8924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:54:41.735348 8924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:54:41.743427 8924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:54:41.743534 8924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:54:41.743708 8924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.130
I20260925 04:54:41.753490 8924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:33995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:44813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.130
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:54:41.754797 8924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:54:41.756197 8924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:54:41.769594 8929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:41.769641 8930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:41.769670 8932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:54:41.770478 8924 server_base.cc:1034] running on GCE node
I20260925 04:54:41.771098 8924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:54:41.772121 8924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:54:41.773384 8924 hybrid_clock.cc:648] HybridClock initialized: now 1790312081773339 us; error 37 us; skew 500 ppm
I20260925 04:54:41.773728 8924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:54:41 dist-test-slave-xzjr krb5kdc[8684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312081, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:54:41.776942 8924 init.cc:377] Logged in from keytab as kudu/127.6.29.130@KRBTEST.COM (short username kudu)
I20260925 04:54:41.778499 8924 webserver.cc:460] Webserver started at http://127.6.29.130:43245/ using document root <none> and password file <none>
I20260925 04:54:41.779227 8924 fs_manager.cc:359] Metadata directory not provided
I20260925 04:54:41.779383 8924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:54:41.779647 8924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:54:41.782579 8924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-1/data/instance:
uuid: "07d97f090a084c2aba7281aa4753b1e4"
format_stamp: "Formatted at 2026-09-25 04:54:41 on dist-test-slave-xzjr"
server_key: "dba72fadb21e3375373bf5edbe51c7f0"
server_key_iv: "bf08388d8500d36c9cb7b61aeed00943"
server_key_version: "encryptionkey@0"
I20260925 04:54:41.783337 8924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance:
uuid: "07d97f090a084c2aba7281aa4753b1e4"
format_stamp: "Formatted at 2026-09-25 04:54:41 on dist-test-slave-xzjr"
server_key: "dba72fadb21e3375373bf5edbe51c7f0"
server_key_iv: "bf08388d8500d36c9cb7b61aeed00943"
server_key_version: "encryptionkey@0"
I20260925 04:54:41.788546 8924 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 04:54:41.792119 8939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:41.793497 8924 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 04:54:41.793684 8924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-1/wal
uuid: "07d97f090a084c2aba7281aa4753b1e4"
format_stamp: "Formatted at 2026-09-25 04:54:41 on dist-test-slave-xzjr"
server_key: "dba72fadb21e3375373bf5edbe51c7f0"
server_key_iv: "bf08388d8500d36c9cb7b61aeed00943"
server_key_version: "encryptionkey@0"
I20260925 04:54:41.793895 8924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:54:41.806051 8924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:54:41.809999 8924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:54:41.810340 8924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:54:41.811273 8924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:54:41.812709 8924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:54:41.812781 8924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:41.812950 8924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:54:41.813002 8924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:41.845742 8924 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.130:46347
I20260925 04:54:41.845791 9051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.130:46347 every 8 connection(s)
I20260925 04:54:41.847054 8924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
I20260925 04:54:41.850188 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 8924
I20260925 04:54:41.850369 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance
I20260925 04:54:41.851017 6262 external_mini_cluster.cc:1428] Setting key f18d05879834195f1d11dfc7947bedda
Sep 25 04:54:41 dist-test-slave-xzjr krb5kdc[8684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312081, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.29.131@KRBTEST.COM; defaulting to no policy
I20260925 04:54:41.867894 8920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:41.849632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:43139 (local address 127.6.29.190:44813)
0925 04:54:41.849802 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:41.849808 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:41.851371 (+ 1563us) server_negotiation.cc:407] Connection header received
0925 04:54:41.853490 (+ 2119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:41.853495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:41.853593 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:41.853702 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:41.855216 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:41.855980 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:41.857610 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:41.858031 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:41.858165 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:41.861787 (+ 3622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:41.861792 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:41.861795 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:41.861826 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:41.864596 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:41.865448 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:41.865455 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:41.865458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:41.865594 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:41.866029 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:41.866036 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:41.866041 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:41.866348 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:41.866503 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:41.867277 (+ 774us) server_negotiation.cc:299] Negotiation successful
0925 04:54:41.867504 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 04:54:41.868919 9054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:41.850285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:44813 (local address 127.6.29.130:43139)
0925 04:54:41.851204 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:41.851256 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:41.853222 (+ 1966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:41.853951 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:41.853959 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:41.854470 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:54:41.855046 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:41.855062 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:41.856124 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:41.856136 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:41.857465 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:41.857476 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:41.858250 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:41.858257 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:41.858479 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:41.859194 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:54:41.859215 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:54:41.861554 (+ 2339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:54:41.864762 (+ 3208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:41.864771 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:41.864791 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:41.865325 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:41.865718 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:41.865723 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:41.865727 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:41.865912 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:41.866488 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:54:41.866496 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:54:41.866913 (+ 417us) client_negotiation.cc:769] Sending connection context
0925 04:54:41.867195 (+ 282us) client_negotiation.cc:240] Negotiation successful
0925 04:54:41.867737 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":396,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.131@KRBTEST.COM" created.
I20260925 04:54:41.870564 9052 heartbeater.cc:347] Connected to a master server at 127.6.29.190:44813
I20260925 04:54:41.872044 9052 heartbeater.cc:464] Registering TS with master...
I20260925 04:54:41.872833 9052 heartbeater.cc:511] Master 127.6.29.190:44813 requested a full tablet report, sending...
I20260925 04:54:41.874400 8729 ts_manager.cc:194] Registered new tserver with Master: 07d97f090a084c2aba7281aa4753b1e4 (127.6.29.130:46347)
I20260925 04:54:41.875209 8729 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.130@KRBTEST.COM'} at 127.6.29.130:43139
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab.
Entry for principal kudu/127.6.29.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab.
I20260925 04:54:41.894743 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.131:0
--local_ip_for_outbound_sockets=127.6.29.131
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:44813
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:33995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:54:42.231155 9057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:54:42.231446 9057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:54:42.231608 9057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:54:42.238967 9057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:54:42.239081 9057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:54:42.239292 9057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.131
I20260925 04:54:42.249101 9057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:33995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:44813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.131
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:54:42.250452 9057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:54:42.251823 9057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:54:42.265393 9062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:42.265462 9063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:42.265427 9065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:54:42.266227 9057 server_base.cc:1034] running on GCE node
I20260925 04:54:42.266867 9057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:54:42.267926 9057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:54:42.269150 9057 hybrid_clock.cc:648] HybridClock initialized: now 1790312082269091 us; error 52 us; skew 500 ppm
I20260925 04:54:42.269479 9057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:54:42 dist-test-slave-xzjr krb5kdc[8684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312082, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:54:42.272568 9057 init.cc:377] Logged in from keytab as kudu/127.6.29.131@KRBTEST.COM (short username kudu)
I20260925 04:54:42.273993 9057 webserver.cc:460] Webserver started at http://127.6.29.131:45621/ using document root <none> and password file <none>
I20260925 04:54:42.274645 9057 fs_manager.cc:359] Metadata directory not provided
I20260925 04:54:42.274741 9057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:54:42.275003 9057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:54:42.277959 9057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-2/data/instance:
uuid: "a76574e918b144e387b59a0f77a32cf6"
format_stamp: "Formatted at 2026-09-25 04:54:42 on dist-test-slave-xzjr"
server_key: "95da49906cdc6fa59d19a8aa85e10dbf"
server_key_iv: "2b8fc2ab889b0e2a7dd5c893259a0016"
server_key_version: "encryptionkey@0"
I20260925 04:54:42.278859 9057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance:
uuid: "a76574e918b144e387b59a0f77a32cf6"
format_stamp: "Formatted at 2026-09-25 04:54:42 on dist-test-slave-xzjr"
server_key: "95da49906cdc6fa59d19a8aa85e10dbf"
server_key_iv: "2b8fc2ab889b0e2a7dd5c893259a0016"
server_key_version: "encryptionkey@0"
I20260925 04:54:42.284406 9057 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260925 04:54:42.288074 9072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:42.289458 9057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:54:42.289639 9057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-2/wal
uuid: "a76574e918b144e387b59a0f77a32cf6"
format_stamp: "Formatted at 2026-09-25 04:54:42 on dist-test-slave-xzjr"
server_key: "95da49906cdc6fa59d19a8aa85e10dbf"
server_key_iv: "2b8fc2ab889b0e2a7dd5c893259a0016"
server_key_version: "encryptionkey@0"
I20260925 04:54:42.289856 9057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:54:42.315754 9057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:54:42.320267 9057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:54:42.320580 9057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:54:42.321589 9057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:54:42.322978 9057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:54:42.323079 9057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:42.323220 9057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:54:42.323289 9057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:42.357311 9057 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.131:35911
I20260925 04:54:42.357350 9184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.131:35911 every 8 connection(s)
I20260925 04:54:42.358634 9057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
I20260925 04:54:42.364403 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 9057
I20260925 04:54:42.364588 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance
I20260925 04:54:42.365027 6262 external_mini_cluster.cc:1428] Setting key bff063ba46f6458fb7338280afcb2795
Sep 25 04:54:42 dist-test-slave-xzjr krb5kdc[8684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312082, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:54:42.378247 8920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:42.361127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:45281 (local address 127.6.29.190:44813)
0925 04:54:42.361318 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:42.361323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:42.362506 (+ 1183us) server_negotiation.cc:407] Connection header received
0925 04:54:42.364412 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:42.364418 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:42.364546 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:42.364677 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:42.366429 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:42.367400 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:42.369157 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:42.369562 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:42.369658 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:42.372964 (+ 3306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:42.372969 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:42.372973 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:42.373004 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:42.375490 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:42.376217 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:42.376221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:42.376223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:42.376341 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:42.376733 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:42.376736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:42.376738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:42.376992 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:42.377111 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:42.377785 (+ 674us) server_negotiation.cc:299] Negotiation successful
0925 04:54:42.377944 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 04:54:42.379645 9187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:42.361616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:44813 (local address 127.6.29.131:45281)
0925 04:54:42.362333 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:42.362370 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:42.364139 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:42.364914 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:42.364923 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:42.365500 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:54:42.366259 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:42.366296 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:42.367559 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:42.367563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:42.369030 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:42.369042 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:42.369739 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:42.369744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:42.369933 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:42.370618 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:54:42.370640 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:54:42.372786 (+ 2146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:54:42.375615 (+ 2829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:42.375622 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:42.375635 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:42.376108 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:42.376432 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:42.376435 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:42.376437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:42.376601 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:42.377098 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:54:42.377104 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:54:42.377425 (+ 321us) client_negotiation.cc:769] Sending connection context
0925 04:54:42.377653 (+ 228us) client_negotiation.cc:240] Negotiation successful
0925 04:54:42.378036 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":280,"threads_started":1}
I20260925 04:54:42.381227 9185 heartbeater.cc:347] Connected to a master server at 127.6.29.190:44813
I20260925 04:54:42.381568 9185 heartbeater.cc:464] Registering TS with master...
I20260925 04:54:42.382333 9185 heartbeater.cc:511] Master 127.6.29.190:44813 requested a full tablet report, sending...
I20260925 04:54:42.383755 8728 ts_manager.cc:194] Registered new tserver with Master: a76574e918b144e387b59a0f77a32cf6 (127.6.29.131:35911)
I20260925 04:54:42.384686 8728 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.131@KRBTEST.COM'} at 127.6.29.131:45281
I20260925 04:54:42.387372 8918 heartbeater.cc:503] Master 127.6.29.190:44813 was elected leader, sending a full tablet report...
I20260925 04:54:42.393102 6262 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 04:54:42.424067 8920 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:54:42.421666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42130 (local address 127.6.29.190:44813)
0925 04:54:42.421909 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:42.421919 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:42.422250 (+ 331us) server_negotiation.cc:407] Connection header received
0925 04:54:42.422403 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:42.422407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:42.422472 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:42.422557 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:42.423770 (+ 1213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42130: BlockingRecv error: recv got EOF from 127.0.0.1:42130 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 04:54:42.426927 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 8790
W20260925 04:54:42.429736 6262 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 8790: Not authorized: Client connection negotiation failed: client connection to 127.6.29.129:42063: server requires authentication, but client does not have Kerberos credentials available
I20260925 04:54:42.429925 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 8790
I20260925 04:54:42.447047 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 8924
W20260925 04:54:42.449772 6262 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 8924: Not authorized: Client connection negotiation failed: client connection to 127.6.29.130:46347: server requires authentication, but client does not have Kerberos credentials available
I20260925 04:54:42.449934 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 8924
I20260925 04:54:42.465377 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 9057
W20260925 04:54:42.468569 6262 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 9057: Not authorized: Client connection negotiation failed: client connection to 127.6.29.131:35911: server requires authentication, but client does not have Kerberos credentials available
I20260925 04:54:42.468742 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 9057
I20260925 04:54:42.484508 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 8697
I20260925 04:54:42.485773 8759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:54:42.588830 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 8697
2026-09-25T04:54:42Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3564 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:54:42 dist-test-slave-xzjr krb5kdc[9202](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:54:42 dist-test-slave-xzjr krb5kdc[9202](info): set up 2 sockets
Sep 25 04:54:42 dist-test-slave-xzjr krb5kdc[9202](info): commencing operation
krb5kdc: starting...
W20260925 04:54:43.680644 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 04:54:43 dist-test-slave-xzjr krb5kdc[9202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:54:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:54:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.29.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
I20260925 04:54:43.883512 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:40933
--webserver_interface=127.6.29.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:45197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:40933
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:54:44.211179 9215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:54:44.211485 9215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:54:44.211654 9215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:54:44.221565 9215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:54:44.221728 9215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:54:44.221807 9215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:54:44.221886 9215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:54:44.221952 9215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:54:44.234946 9215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:45197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:40933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:40933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:54:44.236380 9215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:54:44.237721 9215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:54:44.249781 9220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:44.249748 9221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:44.249723 9223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:54:44.250608 9215 server_base.cc:1034] running on GCE node
I20260925 04:54:44.251405 9215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:54:44.252923 9215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:54:44.254119 9215 hybrid_clock.cc:648] HybridClock initialized: now 1790312084254080 us; error 30 us; skew 500 ppm
I20260925 04:54:44.254495 9215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:54:44 dist-test-slave-xzjr krb5kdc[9202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312084, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:54:44.257704 9215 init.cc:377] Logged in from keytab as kudu/127.6.29.190@KRBTEST.COM (short username kudu)
I20260925 04:54:44.259186 9215 webserver.cc:460] Webserver started at http://127.6.29.190:36753/ using document root <none> and password file <none>
I20260925 04:54:44.259940 9215 fs_manager.cc:359] Metadata directory not provided
I20260925 04:54:44.260077 9215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:54:44.260392 9215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:54:44.263456 9215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "9e62bd8d26724b8aac41af045393981e"
format_stamp: "Formatted at 2026-09-25 04:54:44 on dist-test-slave-xzjr"
server_key: "997ea02c9eab7ac07abd53af591c1cd4"
server_key_iv: "5b615f9e072d23d8e61cdbdc63293a35"
server_key_version: "encryptionkey@0"
I20260925 04:54:44.264303 9215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "9e62bd8d26724b8aac41af045393981e"
format_stamp: "Formatted at 2026-09-25 04:54:44 on dist-test-slave-xzjr"
server_key: "997ea02c9eab7ac07abd53af591c1cd4"
server_key_iv: "5b615f9e072d23d8e61cdbdc63293a35"
server_key_version: "encryptionkey@0"
I20260925 04:54:44.269410 9215 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 04:54:44.272955 9230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:44.274281 9215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:54:44.274466 9215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "9e62bd8d26724b8aac41af045393981e"
format_stamp: "Formatted at 2026-09-25 04:54:44 on dist-test-slave-xzjr"
server_key: "997ea02c9eab7ac07abd53af591c1cd4"
server_key_iv: "5b615f9e072d23d8e61cdbdc63293a35"
server_key_version: "encryptionkey@0"
I20260925 04:54:44.274741 9215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:54:44.294970 9215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:54:44.299106 9215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:54:44.299422 9215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:54:44.317838 9215 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:40933
I20260925 04:54:44.317874 9281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:40933 every 8 connection(s)
I20260925 04:54:44.319267 9215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:54:44.323022 9282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:54:44.324262 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 9215
I20260925 04:54:44.324503 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 04:54:44.325129 6262 external_mini_cluster.cc:1428] Setting key b3548a06b48150ea50977985733636fe
I20260925 04:54:44.336898 9282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e62bd8d26724b8aac41af045393981e: Bootstrap starting.
Sep 25 04:54:44 dist-test-slave-xzjr krb5kdc[9202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:54:44.340874 9282 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e62bd8d26724b8aac41af045393981e: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:44.342106 9282 log.cc:824] T 00000000000000000000000000000000 P 9e62bd8d26724b8aac41af045393981e: Log is configured to *not* fsync() on all Append() calls
I20260925 04:54:44.345503 9282 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e62bd8d26724b8aac41af045393981e: No bootstrap required, opened a new log
I20260925 04:54:44.348832 9285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:44.327180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49588 (local address 127.6.29.190:40933)
0925 04:54:44.328066 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:44.328078 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:44.328118 (+ 40us) server_negotiation.cc:407] Connection header received
0925 04:54:44.329654 (+ 1536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:44.329679 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:44.330081 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:44.330410 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:44.332232 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:44.333529 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:44.335301 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:44.336069 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:44.336202 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:44.340880 (+ 4678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:44.340888 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:44.340903 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:44.340971 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:44.344462 (+ 3491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:44.345326 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:44.345332 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:44.345338 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:44.345508 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:44.345904 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:44.345910 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:44.345913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:44.346453 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:44.346762 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:44.347247 (+ 485us) server_negotiation.cc:299] Negotiation successful
0925 04:54:44.347686 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":425,"threads_started":1}
I20260925 04:54:44.352996 6262 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:54:44.355031 9282 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e62bd8d26724b8aac41af045393981e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e62bd8d26724b8aac41af045393981e" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40933 } }
I20260925 04:54:44.355491 9282 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e62bd8d26724b8aac41af045393981e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:44.355602 9282 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e62bd8d26724b8aac41af045393981e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e62bd8d26724b8aac41af045393981e, State: Initialized, Role: FOLLOWER
I20260925 04:54:44.356297 9282 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e62bd8d26724b8aac41af045393981e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e62bd8d26724b8aac41af045393981e" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40933 } }
I20260925 04:54:44.356588 9282 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e62bd8d26724b8aac41af045393981e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:54:44.356729 9282 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e62bd8d26724b8aac41af045393981e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:54:44.356916 9282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e62bd8d26724b8aac41af045393981e [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:44.360581 9282 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e62bd8d26724b8aac41af045393981e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e62bd8d26724b8aac41af045393981e" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40933 } }
I20260925 04:54:44.361265 9282 leader_election.cc:302] T 00000000000000000000000000000000 P 9e62bd8d26724b8aac41af045393981e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e62bd8d26724b8aac41af045393981e; no voters:
I20260925 04:54:44.362000 9282 leader_election.cc:288] T 00000000000000000000000000000000 P 9e62bd8d26724b8aac41af045393981e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:54:44.362123 9288 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e62bd8d26724b8aac41af045393981e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:54:44.365427 9288 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e62bd8d26724b8aac41af045393981e [term 1 LEADER]: Becoming Leader. State: Replica: 9e62bd8d26724b8aac41af045393981e, State: Running, Role: LEADER
I20260925 04:54:44.366024 9288 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e62bd8d26724b8aac41af045393981e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e62bd8d26724b8aac41af045393981e" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40933 } }
I20260925 04:54:44.366573 9282 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e62bd8d26724b8aac41af045393981e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:54:44.368884 9289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e62bd8d26724b8aac41af045393981e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e62bd8d26724b8aac41af045393981e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e62bd8d26724b8aac41af045393981e" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40933 } } }
I20260925 04:54:44.369248 9289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e62bd8d26724b8aac41af045393981e [sys.catalog]: This master's current role is: LEADER
I20260925 04:54:44.369660 9290 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e62bd8d26724b8aac41af045393981e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e62bd8d26724b8aac41af045393981e. Latest consensus state: current_term: 1 leader_uuid: "9e62bd8d26724b8aac41af045393981e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e62bd8d26724b8aac41af045393981e" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40933 } } }
I20260925 04:54:44.369951 9290 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e62bd8d26724b8aac41af045393981e [sys.catalog]: This master's current role is: LEADER
I20260925 04:54:44.382100 9294 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:54:44.387240 9294 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:54:44.410737 9294 catalog_manager.cc:1347] Generated new cluster ID: 8e0af57ddbee4c9aa0916b281b9fea20
I20260925 04:54:44.410881 9294 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:54:44.435212 9294 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:54:44.436617 9285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:44.422178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49596 (local address 127.6.29.190:40933)
0925 04:54:44.422436 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:44.422446 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:44.426600 (+ 4154us) server_negotiation.cc:407] Connection header received
0925 04:54:44.427113 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:44.427120 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:44.427204 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:44.427280 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:44.428796 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:44.430240 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:44.432190 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:44.432925 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:44.433023 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:44.434772 (+ 1749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:44.434788 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:44.434794 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:54:44.435673 (+ 879us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:44.435751 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:54:44.435816 (+ 65us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:44.435926 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:44.436140 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 04:54:44.436240 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":255,"server-negotiator.queue_time_us":56}
I20260925 04:54:44.437067 9294 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:54:44.457180 9294 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e62bd8d26724b8aac41af045393981e: Generated new TSK 0
I20260925 04:54:44.458149 9294 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:54:44.468851 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 9215
I20260925 04:54:44.470479 9277 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:54:44.564929 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 9215
2026-09-25T04:54:44Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1966 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:54:44 dist-test-slave-xzjr krb5kdc[9316](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:54:44 dist-test-slave-xzjr krb5kdc[9316](info): set up 2 sockets
Sep 25 04:54:44 dist-test-slave-xzjr krb5kdc[9316](info): commencing operation
krb5kdc: starting...
W20260925 04:54:45.668641 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 04:54:45 dist-test-slave-xzjr krb5kdc[9316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:54:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:54:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.29.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
I20260925 04:54:45.871682 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:33611
--webserver_interface=127.6.29.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:41411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:33611
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:54:46.197705 9329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:54:46.198123 9329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:54:46.198302 9329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:54:46.205813 9329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:54:46.205936 9329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:54:46.206029 9329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:54:46.206121 9329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:54:46.206189 9329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:54:46.216468 9329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:41411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:33611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:33611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:54:46.217914 9329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:54:46.219971 9329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:54:46.233697 9334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:46.234064 9337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:46.234350 9335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:54:46.235144 9329 server_base.cc:1034] running on GCE node
I20260925 04:54:46.236110 9329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:54:46.237583 9329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:54:46.238832 9329 hybrid_clock.cc:648] HybridClock initialized: now 1790312086238763 us; error 85 us; skew 500 ppm
I20260925 04:54:46.239209 9329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:54:46 dist-test-slave-xzjr krb5kdc[9316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312086, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:54:46.242427 9329 init.cc:377] Logged in from keytab as kudu/127.6.29.190@KRBTEST.COM (short username kudu)
I20260925 04:54:46.244002 9329 webserver.cc:460] Webserver started at http://127.6.29.190:38615/ using document root <none> and password file <none>
I20260925 04:54:46.244675 9329 fs_manager.cc:359] Metadata directory not provided
I20260925 04:54:46.244772 9329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:54:46.245024 9329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:54:46.247989 9329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "ed7c9d9eeb9c4d539ca14eca125cc97b"
format_stamp: "Formatted at 2026-09-25 04:54:46 on dist-test-slave-xzjr"
server_key: "3ff47f95436def933dd986c053519f3f"
server_key_iv: "69a42185e43bf20d42b6c3d939bbcf2c"
server_key_version: "encryptionkey@0"
I20260925 04:54:46.248708 9329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "ed7c9d9eeb9c4d539ca14eca125cc97b"
format_stamp: "Formatted at 2026-09-25 04:54:46 on dist-test-slave-xzjr"
server_key: "3ff47f95436def933dd986c053519f3f"
server_key_iv: "69a42185e43bf20d42b6c3d939bbcf2c"
server_key_version: "encryptionkey@0"
I20260925 04:54:46.253865 9329 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 04:54:46.257396 9344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:46.258919 9329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260925 04:54:46.259064 9329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "ed7c9d9eeb9c4d539ca14eca125cc97b"
format_stamp: "Formatted at 2026-09-25 04:54:46 on dist-test-slave-xzjr"
server_key: "3ff47f95436def933dd986c053519f3f"
server_key_iv: "69a42185e43bf20d42b6c3d939bbcf2c"
server_key_version: "encryptionkey@0"
I20260925 04:54:46.259243 9329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:54:46.290026 9329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:54:46.294554 9329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:54:46.294796 9329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:54:46.312994 9329 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:33611
I20260925 04:54:46.313026 9395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:33611 every 8 connection(s)
I20260925 04:54:46.314388 9329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:54:46.318146 9396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:54:46.321033 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 9329
I20260925 04:54:46.321280 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 04:54:46.322067 6262 external_mini_cluster.cc:1428] Setting key 15de55bf6947c5b917f3acea797bb515
I20260925 04:54:46.330756 9396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed7c9d9eeb9c4d539ca14eca125cc97b: Bootstrap starting.
I20260925 04:54:46.334640 9396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed7c9d9eeb9c4d539ca14eca125cc97b: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:46.336066 9396 log.cc:824] T 00000000000000000000000000000000 P ed7c9d9eeb9c4d539ca14eca125cc97b: Log is configured to *not* fsync() on all Append() calls
Sep 25 04:54:46 dist-test-slave-xzjr krb5kdc[9316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:54:46.339200 9396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed7c9d9eeb9c4d539ca14eca125cc97b: No bootstrap required, opened a new log
I20260925 04:54:46.345911 9399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:46.324276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58152 (local address 127.6.29.190:33611)
0925 04:54:46.325051 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:46.325063 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:46.325105 (+ 42us) server_negotiation.cc:407] Connection header received
0925 04:54:46.326729 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:46.326751 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:46.327168 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:46.327512 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:46.329217 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:46.330544 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:46.332318 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:46.332993 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:46.333119 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:46.338114 (+ 4995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:46.338122 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:46.338137 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:46.338183 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:46.341512 (+ 3329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:46.342291 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:46.342298 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:46.342305 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:46.342475 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:46.342903 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:46.342912 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:46.342915 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:46.343486 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:46.343851 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:46.344350 (+ 499us) server_negotiation.cc:299] Negotiation successful
0925 04:54:46.344807 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":375,"threads_started":1}
I20260925 04:54:46.347259 9396 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed7c9d9eeb9c4d539ca14eca125cc97b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed7c9d9eeb9c4d539ca14eca125cc97b" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33611 } }
I20260925 04:54:46.347793 9396 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed7c9d9eeb9c4d539ca14eca125cc97b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:46.347905 9396 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed7c9d9eeb9c4d539ca14eca125cc97b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed7c9d9eeb9c4d539ca14eca125cc97b, State: Initialized, Role: FOLLOWER
I20260925 04:54:46.348613 9396 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed7c9d9eeb9c4d539ca14eca125cc97b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed7c9d9eeb9c4d539ca14eca125cc97b" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33611 } }
I20260925 04:54:46.348953 9396 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed7c9d9eeb9c4d539ca14eca125cc97b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:54:46.349112 9396 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed7c9d9eeb9c4d539ca14eca125cc97b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:54:46.349313 9396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed7c9d9eeb9c4d539ca14eca125cc97b [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:46.351456 9396 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed7c9d9eeb9c4d539ca14eca125cc97b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed7c9d9eeb9c4d539ca14eca125cc97b" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33611 } }
I20260925 04:54:46.352028 9396 leader_election.cc:302] T 00000000000000000000000000000000 P ed7c9d9eeb9c4d539ca14eca125cc97b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed7c9d9eeb9c4d539ca14eca125cc97b; no voters:
I20260925 04:54:46.352650 9396 leader_election.cc:288] T 00000000000000000000000000000000 P ed7c9d9eeb9c4d539ca14eca125cc97b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:54:46.352828 9401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed7c9d9eeb9c4d539ca14eca125cc97b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:54:46.353528 9401 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed7c9d9eeb9c4d539ca14eca125cc97b [term 1 LEADER]: Becoming Leader. State: Replica: ed7c9d9eeb9c4d539ca14eca125cc97b, State: Running, Role: LEADER
I20260925 04:54:46.354182 9401 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed7c9d9eeb9c4d539ca14eca125cc97b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed7c9d9eeb9c4d539ca14eca125cc97b" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33611 } }
I20260925 04:54:46.354915 9396 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed7c9d9eeb9c4d539ca14eca125cc97b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:54:46.357429 9403 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed7c9d9eeb9c4d539ca14eca125cc97b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed7c9d9eeb9c4d539ca14eca125cc97b. Latest consensus state: current_term: 1 leader_uuid: "ed7c9d9eeb9c4d539ca14eca125cc97b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed7c9d9eeb9c4d539ca14eca125cc97b" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33611 } } }
I20260925 04:54:46.357852 9403 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed7c9d9eeb9c4d539ca14eca125cc97b [sys.catalog]: This master's current role is: LEADER
I20260925 04:54:46.357865 9402 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed7c9d9eeb9c4d539ca14eca125cc97b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed7c9d9eeb9c4d539ca14eca125cc97b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed7c9d9eeb9c4d539ca14eca125cc97b" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33611 } } }
I20260925 04:54:46.358393 9402 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed7c9d9eeb9c4d539ca14eca125cc97b [sys.catalog]: This master's current role is: LEADER
I20260925 04:54:46.360595 9410 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:54:46.365166 9410 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:54:46.373965 9410 catalog_manager.cc:1347] Generated new cluster ID: 8834dd5e4612488e9d2a71cd7ab7ed61
I20260925 04:54:46.374104 9410 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:54:46.413046 9410 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:54:46.414984 9410 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:54:46.428301 9410 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed7c9d9eeb9c4d539ca14eca125cc97b: Generated new TSK 0
I20260925 04:54:46.429384 9410 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.29.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab.
Entry for principal kudu/127.6.29.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab.
I20260925 04:54:46.480376 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.129:0
--local_ip_for_outbound_sockets=127.6.29.129
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:33611
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:41411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:54:46.820183 9422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:54:46.820541 9422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:54:46.820719 9422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:54:46.828222 9422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:54:46.828400 9422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:54:46.828593 9422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.129
I20260925 04:54:46.838722 9422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:41411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:33611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.129
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:54:46.840350 9422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:54:46.841704 9422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:54:46.856582 9428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:46.856608 9430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:46.857362 9427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:54:46.857970 9422 server_base.cc:1034] running on GCE node
I20260925 04:54:46.858661 9422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:54:46.859742 9422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:54:46.861009 9422 hybrid_clock.cc:648] HybridClock initialized: now 1790312086860919 us; error 103 us; skew 500 ppm
I20260925 04:54:46.861366 9422 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:54:46 dist-test-slave-xzjr krb5kdc[9316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312086, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:54:46.864984 9422 init.cc:377] Logged in from keytab as kudu/127.6.29.129@KRBTEST.COM (short username kudu)
I20260925 04:54:46.866495 9422 webserver.cc:460] Webserver started at http://127.6.29.129:33651/ using document root <none> and password file <none>
I20260925 04:54:46.867233 9422 fs_manager.cc:359] Metadata directory not provided
I20260925 04:54:46.867383 9422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:54:46.867705 9422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:54:46.870806 9422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-0/data/instance:
uuid: "87f4be63fb194f2a987a863b6c1479bb"
format_stamp: "Formatted at 2026-09-25 04:54:46 on dist-test-slave-xzjr"
server_key: "5e98de206b8c884938b0fa4d16a7df6e"
server_key_iv: "520cdf52e16b8b7b9cde178534547d04"
server_key_version: "encryptionkey@0"
I20260925 04:54:46.871717 9422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance:
uuid: "87f4be63fb194f2a987a863b6c1479bb"
format_stamp: "Formatted at 2026-09-25 04:54:46 on dist-test-slave-xzjr"
server_key: "5e98de206b8c884938b0fa4d16a7df6e"
server_key_iv: "520cdf52e16b8b7b9cde178534547d04"
server_key_version: "encryptionkey@0"
I20260925 04:54:46.877094 9422 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 04:54:46.880816 9437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:46.882239 9422 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:54:46.882431 9422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-0/wal
uuid: "87f4be63fb194f2a987a863b6c1479bb"
format_stamp: "Formatted at 2026-09-25 04:54:46 on dist-test-slave-xzjr"
server_key: "5e98de206b8c884938b0fa4d16a7df6e"
server_key_iv: "520cdf52e16b8b7b9cde178534547d04"
server_key_version: "encryptionkey@0"
I20260925 04:54:46.882650 9422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:54:46.895718 9422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:54:46.900295 9422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:54:46.900621 9422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:54:46.901611 9422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:54:46.902953 9422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:54:46.903059 9422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:46.903216 9422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:54:46.903294 9422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:46.937772 9422 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.129:33789
I20260925 04:54:46.937821 9549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.129:33789 every 8 connection(s)
I20260925 04:54:46.939152 9422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
I20260925 04:54:46.941843 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 9422
I20260925 04:54:46.942086 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance
I20260925 04:54:46.942751 6262 external_mini_cluster.cc:1428] Setting key 74b2f40a41a6a263129ad0673c8df544
Sep 25 04:54:46 dist-test-slave-xzjr krb5kdc[9316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312086, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.29.130@KRBTEST.COM; defaulting to no policy
I20260925 04:54:46.961537 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:46.941839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:45403 (local address 127.6.29.190:33611)
0925 04:54:46.942330 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:46.942337 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:46.943549 (+ 1212us) server_negotiation.cc:407] Connection header received
0925 04:54:46.945921 (+ 2372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:46.945926 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:46.945994 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:46.946092 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:46.948428 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:46.949152 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:46.951298 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:46.951703 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:46.951906 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:46.955446 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:46.955451 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:46.955455 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:46.955488 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:46.958449 (+ 2961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:46.959204 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:46.959216 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:46.959219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:46.959341 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:46.959800 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:46.959805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:46.959808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:46.960071 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:46.960225 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:46.960947 (+ 722us) server_negotiation.cc:299] Negotiation successful
0925 04:54:46.961169 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":239,"threads_started":1}
I20260925 04:54:46.962684 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:46.942504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:33611 (local address 127.6.29.129:45403)
0925 04:54:46.943369 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:46.943430 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:46.945587 (+ 2157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:46.946404 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:46.946420 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:46.947288 (+ 868us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:54:46.948210 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:46.948239 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:46.949340 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:46.949348 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:46.951129 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:46.951142 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:46.951996 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:46.952004 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:46.952292 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:46.952999 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:54:46.953021 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:54:46.955223 (+ 2202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:54:46.958613 (+ 3390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:46.958623 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:46.958641 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:46.959073 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:46.959455 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:46.959461 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:46.959464 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:46.959664 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:46.960201 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:54:46.960210 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:54:46.960578 (+ 368us) client_negotiation.cc:769] Sending connection context
0925 04:54:46.960852 (+ 274us) client_negotiation.cc:240] Negotiation successful
0925 04:54:46.961425 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":354,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.130@KRBTEST.COM" created.
I20260925 04:54:46.964711 9550 heartbeater.cc:347] Connected to a master server at 127.6.29.190:33611
I20260925 04:54:46.965030 9550 heartbeater.cc:464] Registering TS with master...
I20260925 04:54:46.965775 9550 heartbeater.cc:511] Master 127.6.29.190:33611 requested a full tablet report, sending...
I20260925 04:54:46.967950 9361 ts_manager.cc:194] Registered new tserver with Master: 87f4be63fb194f2a987a863b6c1479bb (127.6.29.129:33789)
I20260925 04:54:46.969368 9361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.129@KRBTEST.COM'} at 127.6.29.129:45403
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab.
Entry for principal kudu/127.6.29.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab.
I20260925 04:54:46.988075 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.130:0
--local_ip_for_outbound_sockets=127.6.29.130
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:33611
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:41411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:54:47.321993 9556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:54:47.322351 9556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:54:47.322520 9556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:54:47.330246 9556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:54:47.330418 9556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:54:47.330614 9556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.130
I20260925 04:54:47.340694 9556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:41411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:33611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.130
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:54:47.342167 9556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:54:47.343521 9556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:54:47.356513 9561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:47.356536 9562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:47.357056 9564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:54:47.357427 9556 server_base.cc:1034] running on GCE node
I20260925 04:54:47.358076 9556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:54:47.359117 9556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:54:47.360339 9556 hybrid_clock.cc:648] HybridClock initialized: now 1790312087360285 us; error 51 us; skew 500 ppm
I20260925 04:54:47.360667 9556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:54:47 dist-test-slave-xzjr krb5kdc[9316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312087, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:54:47.364157 9556 init.cc:377] Logged in from keytab as kudu/127.6.29.130@KRBTEST.COM (short username kudu)
I20260925 04:54:47.365694 9556 webserver.cc:460] Webserver started at http://127.6.29.130:45825/ using document root <none> and password file <none>
I20260925 04:54:47.366415 9556 fs_manager.cc:359] Metadata directory not provided
I20260925 04:54:47.366545 9556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:54:47.366869 9556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:54:47.369877 9556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-1/data/instance:
uuid: "b2deaddf2a114e75837ef19e8ca75d2d"
format_stamp: "Formatted at 2026-09-25 04:54:47 on dist-test-slave-xzjr"
server_key: "a65d19ee0a6ebecc1d84bb3e4e322703"
server_key_iv: "c6edd9902d3fa241a7dd189d5bf269f6"
server_key_version: "encryptionkey@0"
I20260925 04:54:47.370692 9556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance:
uuid: "b2deaddf2a114e75837ef19e8ca75d2d"
format_stamp: "Formatted at 2026-09-25 04:54:47 on dist-test-slave-xzjr"
server_key: "a65d19ee0a6ebecc1d84bb3e4e322703"
server_key_iv: "c6edd9902d3fa241a7dd189d5bf269f6"
server_key_version: "encryptionkey@0"
I20260925 04:54:47.376128 9556 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 04:54:47.379892 9571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:47.381641 9556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:54:47.381794 9556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-1/wal
uuid: "b2deaddf2a114e75837ef19e8ca75d2d"
format_stamp: "Formatted at 2026-09-25 04:54:47 on dist-test-slave-xzjr"
server_key: "a65d19ee0a6ebecc1d84bb3e4e322703"
server_key_iv: "c6edd9902d3fa241a7dd189d5bf269f6"
server_key_version: "encryptionkey@0"
I20260925 04:54:47.381979 9556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:54:47.397608 9556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:54:47.401713 9556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:54:47.401964 9556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:54:47.402878 9556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:54:47.404348 9556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:54:47.404425 9556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:47.404510 9556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:54:47.404557 9556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:47.438726 9556 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.130:40881
I20260925 04:54:47.438766 9683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.130:40881 every 8 connection(s)
I20260925 04:54:47.440133 9556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
I20260925 04:54:47.447469 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 9556
I20260925 04:54:47.447691 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance
I20260925 04:54:47.448211 6262 external_mini_cluster.cc:1428] Setting key 8c7733c4204494e637ae911464180d29
Sep 25 04:54:47 dist-test-slave-xzjr krb5kdc[9316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312087, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:54:47.463874 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:47.442978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:50019 (local address 127.6.29.190:33611)
0925 04:54:47.443432 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:47.443437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:47.444456 (+ 1019us) server_negotiation.cc:407] Connection header received
0925 04:54:47.446341 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:47.446347 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:47.446440 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:47.446560 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:47.448694 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:47.449494 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:47.451733 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:47.452207 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:47.452324 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:47.457176 (+ 4852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:47.457184 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:47.457189 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:47.457236 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:47.460162 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:47.461174 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:47.461178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:47.461180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:47.461308 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:47.461840 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:47.461843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:47.461845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:47.462181 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:47.462293 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:47.463268 (+ 975us) server_negotiation.cc:299] Negotiation successful
0925 04:54:47.463573 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 04:54:47.465459 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:47.443488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:33611 (local address 127.6.29.130:50019)
0925 04:54:47.444300 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:47.444338 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:47.446045 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:47.446812 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:47.446821 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:47.447569 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:54:47.448478 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:47.448504 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:47.449641 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:47.449645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:47.451576 (+ 1931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:47.451595 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:47.452457 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:47.452467 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:47.452811 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:47.453558 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:54:47.453592 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:54:47.456892 (+ 3300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:54:47.460313 (+ 3421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:47.460327 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:47.460349 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:47.461019 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:47.461427 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:47.461435 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:47.461442 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:47.461713 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:47.462307 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:54:47.462318 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:54:47.462841 (+ 523us) client_negotiation.cc:769] Sending connection context
0925 04:54:47.463165 (+ 324us) client_negotiation.cc:240] Negotiation successful
0925 04:54:47.463834 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":385,"threads_started":1}
I20260925 04:54:47.467235 9684 heartbeater.cc:347] Connected to a master server at 127.6.29.190:33611
I20260925 04:54:47.467555 9684 heartbeater.cc:464] Registering TS with master...
I20260925 04:54:47.468374 9684 heartbeater.cc:511] Master 127.6.29.190:33611 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.29.131@KRBTEST.COM; defaulting to no policy
I20260925 04:54:47.469967 9361 ts_manager.cc:194] Registered new tserver with Master: b2deaddf2a114e75837ef19e8ca75d2d (127.6.29.130:40881)
I20260925 04:54:47.470846 9361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.130@KRBTEST.COM'} at 127.6.29.130:50019
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab.
Entry for principal kudu/127.6.29.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab.
I20260925 04:54:47.495437 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.131:0
--local_ip_for_outbound_sockets=127.6.29.131
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:33611
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:41411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:54:47.826568 9689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:54:47.826900 9689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:54:47.827047 9689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:54:47.834569 9689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:54:47.834692 9689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:54:47.834874 9689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.131
I20260925 04:54:47.845065 9689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:41411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:33611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.131
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:54:47.846441 9689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:54:47.847790 9689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:54:47.865800 9694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:47.866237 9695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:47.866715 9697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:54:47.867149 9689 server_base.cc:1034] running on GCE node
I20260925 04:54:47.868166 9689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:54:47.869558 9689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:54:47.870833 9689 hybrid_clock.cc:648] HybridClock initialized: now 1790312087870780 us; error 40 us; skew 500 ppm
I20260925 04:54:47.871260 9689 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:54:47 dist-test-slave-xzjr krb5kdc[9316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312087, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:54:47.876085 9689 init.cc:377] Logged in from keytab as kudu/127.6.29.131@KRBTEST.COM (short username kudu)
I20260925 04:54:47.878314 9689 webserver.cc:460] Webserver started at http://127.6.29.131:42315/ using document root <none> and password file <none>
I20260925 04:54:47.879262 9689 fs_manager.cc:359] Metadata directory not provided
I20260925 04:54:47.879408 9689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:54:47.879840 9689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:54:47.884330 9689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-2/data/instance:
uuid: "0f42abc57f054cd2bcaae1ba763df781"
format_stamp: "Formatted at 2026-09-25 04:54:47 on dist-test-slave-xzjr"
server_key: "6f520b6c371be3976fc392ebfc47c852"
server_key_iv: "0e88ecfe4f661f1bce7e9aece27bb6a2"
server_key_version: "encryptionkey@0"
I20260925 04:54:47.885459 9689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance:
uuid: "0f42abc57f054cd2bcaae1ba763df781"
format_stamp: "Formatted at 2026-09-25 04:54:47 on dist-test-slave-xzjr"
server_key: "6f520b6c371be3976fc392ebfc47c852"
server_key_iv: "0e88ecfe4f661f1bce7e9aece27bb6a2"
server_key_version: "encryptionkey@0"
I20260925 04:54:47.892648 9689 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260925 04:54:47.898157 9704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:47.900220 9689 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 04:54:47.900416 9689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-2/wal
uuid: "0f42abc57f054cd2bcaae1ba763df781"
format_stamp: "Formatted at 2026-09-25 04:54:47 on dist-test-slave-xzjr"
server_key: "6f520b6c371be3976fc392ebfc47c852"
server_key_iv: "0e88ecfe4f661f1bce7e9aece27bb6a2"
server_key_version: "encryptionkey@0"
I20260925 04:54:47.900682 9689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:54:47.922782 9689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:54:47.927282 9689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:54:47.927561 9689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:54:47.928524 9689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:54:47.929844 9689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:54:47.929914 9689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:47.929999 9689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:54:47.930043 9689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:47.963656 9689 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.131:38503
I20260925 04:54:47.963800 9816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.131:38503 every 8 connection(s)
I20260925 04:54:47.965126 9689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
I20260925 04:54:47.972390 9550 heartbeater.cc:503] Master 127.6.29.190:33611 was elected leader, sending a full tablet report...
I20260925 04:54:47.974543 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 9689
I20260925 04:54:47.974748 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance
I20260925 04:54:47.975299 6262 external_mini_cluster.cc:1428] Setting key 457821461d31c9bd45e9b8c1d66de278
Sep 25 04:54:47 dist-test-slave-xzjr krb5kdc[9316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312087, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:54:47.986482 9819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:47.967748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:35543 (local address 127.6.29.190:33611)
0925 04:54:47.968254 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:47.968260 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:47.969296 (+ 1036us) server_negotiation.cc:407] Connection header received
0925 04:54:47.971021 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:47.971025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:47.971090 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:47.971191 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:47.972749 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:47.973662 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:47.975968 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:47.976510 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:47.976641 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:47.980504 (+ 3863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:47.980509 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:47.980513 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:47.980543 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:47.983371 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:47.984168 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:47.984174 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:47.984178 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:47.984302 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:47.984722 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:47.984727 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:47.984730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:47.985005 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:47.985156 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:47.985874 (+ 718us) server_negotiation.cc:299] Negotiation successful
0925 04:54:47.986090 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":196,"threads_started":1}
I20260925 04:54:47.987473 9820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:47.968446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:33611 (local address 127.6.29.131:35543)
0925 04:54:47.969127 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:47.969181 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:47.970766 (+ 1585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:47.971393 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:47.971401 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:47.971978 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:54:47.972577 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:47.972606 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:47.973816 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:47.973820 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:47.975820 (+ 2000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:47.975837 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:47.976763 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:47.976773 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:47.977081 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:47.977845 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:54:47.977873 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:54:47.980293 (+ 2420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:54:47.983516 (+ 3223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:47.983525 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:47.983541 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:47.984043 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:47.984414 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:47.984419 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:47.984423 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:47.984603 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:47.985131 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:54:47.985139 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:54:47.985500 (+ 361us) client_negotiation.cc:769] Sending connection context
0925 04:54:47.985775 (+ 275us) client_negotiation.cc:240] Negotiation successful
0925 04:54:47.986357 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":266,"threads_started":1}
I20260925 04:54:47.988941 9817 heartbeater.cc:347] Connected to a master server at 127.6.29.190:33611
I20260925 04:54:47.989282 9817 heartbeater.cc:464] Registering TS with master...
I20260925 04:54:47.990139 9817 heartbeater.cc:511] Master 127.6.29.190:33611 requested a full tablet report, sending...
I20260925 04:54:47.991580 9361 ts_manager.cc:194] Registered new tserver with Master: 0f42abc57f054cd2bcaae1ba763df781 (127.6.29.131:38503)
I20260925 04:54:47.992419 9361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.131@KRBTEST.COM'} at 127.6.29.131:35543
I20260925 04:54:48.002948 6262 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 04:54:48 dist-test-slave-xzjr krb5kdc[9316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312088, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 04:54:48 dist-test-slave-xzjr krb5kdc[9316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312088, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:54:48.051007 9819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:48.037246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58158 (local address 127.6.29.190:33611)
0925 04:54:48.037408 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:48.037413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:48.037932 (+ 519us) server_negotiation.cc:407] Connection header received
0925 04:54:48.038121 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:48.038126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:48.038195 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:48.038291 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:48.039643 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:48.040447 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:48.041675 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:48.042065 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:48.042163 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:48.045977 (+ 3814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:48.045983 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:48.045986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:48.046025 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:48.048713 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:48.049352 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:48.049356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:48.049358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:48.049477 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:48.049826 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:48.049831 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:48.049834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:48.050150 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:48.050323 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:48.050576 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 04:54:48.050786 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260925 04:54:48.051540 9361 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58158
I20260925 04:54:48.053310 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 9422
Sep 25 04:54:48 dist-test-slave-xzjr krb5kdc[9316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312088, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.29.129@KRBTEST.COM
I20260925 04:54:48.070087 9829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:48.054411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37976 (local address 127.6.29.129:33789)
0925 04:54:48.055071 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:48.055081 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:48.055110 (+ 29us) server_negotiation.cc:407] Connection header received
0925 04:54:48.055202 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:48.055210 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:48.055540 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:48.055696 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:48.057362 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:48.058957 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:48.060173 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:48.060746 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:48.060843 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:48.064751 (+ 3908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:48.064764 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:48.064780 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:48.064820 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:48.067536 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:48.068213 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:48.068220 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:48.068222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:48.068350 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:48.068693 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:48.068697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:48.068700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:48.069045 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:48.069284 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:48.069463 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 04:54:48.069747 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":294,"threads_started":1}
W20260925 04:54:48.071336 9545 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37976
W20260925 04:54:48.072614 6262 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 9422: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 04:54:48.072811 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 9422
I20260925 04:54:48.088980 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 9556
Sep 25 04:54:48 dist-test-slave-xzjr krb5kdc[9316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312088, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.29.130@KRBTEST.COM
I20260925 04:54:48.105209 9830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:48.089805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58254 (local address 127.6.29.130:40881)
0925 04:54:48.090326 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:48.090333 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:48.090358 (+ 25us) server_negotiation.cc:407] Connection header received
0925 04:54:48.090418 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:48.090423 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:48.090637 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:48.090825 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:48.092133 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:48.093407 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:48.094706 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:48.095280 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:48.095451 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:48.099663 (+ 4212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:48.099675 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:48.099691 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:48.099821 (+ 130us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:48.102606 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:48.103253 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:48.103261 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:48.103263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:48.103400 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:48.103705 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:48.103708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:48.103710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:48.104132 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:48.104370 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:48.104600 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 04:54:48.104808 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":273,"threads_started":1}
W20260925 04:54:48.106691 9679 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58254
W20260925 04:54:48.107956 6262 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 9556: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 04:54:48.108151 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 9556
I20260925 04:54:48.123716 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 9689
Sep 25 04:54:48 dist-test-slave-xzjr krb5kdc[9316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312088, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.29.131@KRBTEST.COM
I20260925 04:54:48.139673 9831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:48.124751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44984 (local address 127.6.29.131:38503)
0925 04:54:48.125361 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:48.125368 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:48.125396 (+ 28us) server_negotiation.cc:407] Connection header received
0925 04:54:48.125458 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:48.125465 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:48.125671 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:48.125816 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:48.127064 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:48.128211 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:48.129367 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:48.129935 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:48.130031 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:48.134282 (+ 4251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:48.134293 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:48.134308 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:48.134343 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:48.137273 (+ 2930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:48.137876 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:48.137882 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:48.137884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:48.137998 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:48.138293 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:48.138297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:48.138299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:48.138635 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:48.138863 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:48.139073 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 04:54:48.139275 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":300,"threads_started":1}
W20260925 04:54:48.141098 9812 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44984
W20260925 04:54:48.142321 6262 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 9689: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 04:54:48.142470 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 9689
I20260925 04:54:48.157954 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 9329
I20260925 04:54:48.159282 9391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:54:48.264773 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 9329
2026-09-25T04:54:48Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3708 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:54:48 dist-test-slave-xzjr krb5kdc[9835](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:54:48 dist-test-slave-xzjr krb5kdc[9835](info): set up 2 sockets
Sep 25 04:54:48 dist-test-slave-xzjr krb5kdc[9835](info): commencing operation
krb5kdc: starting...
W20260925 04:54:49.352576 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 04:54:49 dist-test-slave-xzjr krb5kdc[9835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:54:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:54:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.29.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
I20260925 04:54:49.559918 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:43343
--webserver_interface=127.6.29.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:33785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:43343
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:54:49.882560 9848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:54:49.882867 9848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:54:49.883028 9848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:54:49.890385 9848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:54:49.890497 9848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:54:49.890592 9848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:54:49.890673 9848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:54:49.890750 9848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:54:49.900722 9848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:33785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:43343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:43343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:54:49.902098 9848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:54:49.903440 9848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:54:49.915300 9853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:49.915300 9856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:49.915359 9854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:54:49.916378 9848 server_base.cc:1034] running on GCE node
I20260925 04:54:49.917086 9848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:54:49.918491 9848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:54:49.919713 9848 hybrid_clock.cc:648] HybridClock initialized: now 1790312089919657 us; error 51 us; skew 500 ppm
I20260925 04:54:49.920077 9848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:54:49 dist-test-slave-xzjr krb5kdc[9835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312089, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:54:49.923170 9848 init.cc:377] Logged in from keytab as kudu/127.6.29.190@KRBTEST.COM (short username kudu)
I20260925 04:54:49.924813 9848 webserver.cc:460] Webserver started at http://127.6.29.190:43615/ using document root <none> and password file <none>
I20260925 04:54:49.925508 9848 fs_manager.cc:359] Metadata directory not provided
I20260925 04:54:49.925647 9848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:54:49.925963 9848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:54:49.928964 9848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "ea836a30d2484e7ab0e4a11a997d2038"
format_stamp: "Formatted at 2026-09-25 04:54:49 on dist-test-slave-xzjr"
server_key: "706b07124440a5124911fc651e0485aa"
server_key_iv: "d929e3ad968877da3cc2c048743ee125"
server_key_version: "encryptionkey@0"
I20260925 04:54:49.929767 9848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "ea836a30d2484e7ab0e4a11a997d2038"
format_stamp: "Formatted at 2026-09-25 04:54:49 on dist-test-slave-xzjr"
server_key: "706b07124440a5124911fc651e0485aa"
server_key_iv: "d929e3ad968877da3cc2c048743ee125"
server_key_version: "encryptionkey@0"
I20260925 04:54:49.935107 9848 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 04:54:49.938748 9863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:49.940277 9848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:54:49.940464 9848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "ea836a30d2484e7ab0e4a11a997d2038"
format_stamp: "Formatted at 2026-09-25 04:54:49 on dist-test-slave-xzjr"
server_key: "706b07124440a5124911fc651e0485aa"
server_key_iv: "d929e3ad968877da3cc2c048743ee125"
server_key_version: "encryptionkey@0"
I20260925 04:54:49.940680 9848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:54:49.960976 9848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:54:49.965619 9848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:54:49.965936 9848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:54:49.984479 9848 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:43343
I20260925 04:54:49.984510 9914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:43343 every 8 connection(s)
I20260925 04:54:49.985982 9848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:54:49.989012 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 9848
I20260925 04:54:49.989223 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 04:54:49.990025 6262 external_mini_cluster.cc:1428] Setting key 5a412d386e6a8f38633bd64f342eaf80
I20260925 04:54:49.990610 9915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:54:50.005477 9915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea836a30d2484e7ab0e4a11a997d2038: Bootstrap starting.
Sep 25 04:54:50 dist-test-slave-xzjr krb5kdc[9835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:54:50.009815 9915 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea836a30d2484e7ab0e4a11a997d2038: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:50.011196 9915 log.cc:824] T 00000000000000000000000000000000 P ea836a30d2484e7ab0e4a11a997d2038: Log is configured to *not* fsync() on all Append() calls
I20260925 04:54:50.014739 9915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea836a30d2484e7ab0e4a11a997d2038: No bootstrap required, opened a new log
I20260925 04:54:50.015908 9918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:49.992187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51808 (local address 127.6.29.190:43343)
0925 04:54:49.992974 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:49.992986 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:49.993027 (+ 41us) server_negotiation.cc:407] Connection header received
0925 04:54:49.995246 (+ 2219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:49.995272 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:49.995711 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:49.996184 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:49.997974 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:49.999309 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:50.001109 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:50.001901 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:50.002059 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:50.006733 (+ 4674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:50.006744 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:50.006766 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:50.006817 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:50.011047 (+ 4230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:50.011891 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:50.011906 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:50.011915 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:50.012089 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:50.012585 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:50.012591 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:50.012594 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:50.013313 (+ 719us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:50.013622 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:50.013998 (+ 376us) server_negotiation.cc:299] Negotiation successful
0925 04:54:50.014435 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":366,"threads_started":1}
I20260925 04:54:50.023355 9915 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea836a30d2484e7ab0e4a11a997d2038 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea836a30d2484e7ab0e4a11a997d2038" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43343 } }
I20260925 04:54:50.023939 9915 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea836a30d2484e7ab0e4a11a997d2038 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:50.024063 9915 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea836a30d2484e7ab0e4a11a997d2038 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea836a30d2484e7ab0e4a11a997d2038, State: Initialized, Role: FOLLOWER
I20260925 04:54:50.024623 9915 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea836a30d2484e7ab0e4a11a997d2038 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea836a30d2484e7ab0e4a11a997d2038" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43343 } }
I20260925 04:54:50.024842 9915 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea836a30d2484e7ab0e4a11a997d2038 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:54:50.024963 9915 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea836a30d2484e7ab0e4a11a997d2038 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:54:50.025120 9915 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea836a30d2484e7ab0e4a11a997d2038 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:50.027256 9915 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea836a30d2484e7ab0e4a11a997d2038 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea836a30d2484e7ab0e4a11a997d2038" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43343 } }
I20260925 04:54:50.027755 9915 leader_election.cc:302] T 00000000000000000000000000000000 P ea836a30d2484e7ab0e4a11a997d2038 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea836a30d2484e7ab0e4a11a997d2038; no voters:
I20260925 04:54:50.028393 9915 leader_election.cc:288] T 00000000000000000000000000000000 P ea836a30d2484e7ab0e4a11a997d2038 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:54:50.028548 9920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea836a30d2484e7ab0e4a11a997d2038 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:54:50.029222 9920 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea836a30d2484e7ab0e4a11a997d2038 [term 1 LEADER]: Becoming Leader. State: Replica: ea836a30d2484e7ab0e4a11a997d2038, State: Running, Role: LEADER
I20260925 04:54:50.029698 9920 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea836a30d2484e7ab0e4a11a997d2038 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea836a30d2484e7ab0e4a11a997d2038" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43343 } }
I20260925 04:54:50.030486 9915 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea836a30d2484e7ab0e4a11a997d2038 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:54:50.033322 9921 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea836a30d2484e7ab0e4a11a997d2038 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea836a30d2484e7ab0e4a11a997d2038" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea836a30d2484e7ab0e4a11a997d2038" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43343 } } }
I20260925 04:54:50.033228 9922 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea836a30d2484e7ab0e4a11a997d2038 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea836a30d2484e7ab0e4a11a997d2038. Latest consensus state: current_term: 1 leader_uuid: "ea836a30d2484e7ab0e4a11a997d2038" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea836a30d2484e7ab0e4a11a997d2038" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43343 } } }
I20260925 04:54:50.033649 9921 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea836a30d2484e7ab0e4a11a997d2038 [sys.catalog]: This master's current role is: LEADER
I20260925 04:54:50.033694 9922 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea836a30d2484e7ab0e4a11a997d2038 [sys.catalog]: This master's current role is: LEADER
I20260925 04:54:50.035264 9927 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:54:50.040881 9927 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:54:50.051020 9927 catalog_manager.cc:1347] Generated new cluster ID: 6b4daf6017f64324ac6778f81afec237
I20260925 04:54:50.051163 9927 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:54:50.060560 9927 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:54:50.062500 9927 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:54:50.071727 9927 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea836a30d2484e7ab0e4a11a997d2038: Generated new TSK 0
I20260925 04:54:50.072917 9927 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.29.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab.
Entry for principal kudu/127.6.29.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab.
I20260925 04:54:50.123366 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.129:0
--local_ip_for_outbound_sockets=127.6.29.129
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:43343
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:33785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:54:50.459522 9941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:54:50.459888 9941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:54:50.460069 9941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:54:50.467666 9941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:54:50.467844 9941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:54:50.468027 9941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.129
I20260925 04:54:50.478034 9941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:33785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:43343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.129
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:54:50.479424 9941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:54:50.480824 9941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:54:50.494232 9946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:50.494280 9947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:50.494321 9949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:54:50.495687 9941 server_base.cc:1034] running on GCE node
I20260925 04:54:50.496377 9941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:54:50.497323 9941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:54:50.498554 9941 hybrid_clock.cc:648] HybridClock initialized: now 1790312090498494 us; error 59 us; skew 500 ppm
I20260925 04:54:50.498895 9941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:54:50 dist-test-slave-xzjr krb5kdc[9835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312090, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:54:50.502128 9941 init.cc:377] Logged in from keytab as kudu/127.6.29.129@KRBTEST.COM (short username kudu)
I20260925 04:54:50.503692 9941 webserver.cc:460] Webserver started at http://127.6.29.129:45571/ using document root <none> and password file <none>
I20260925 04:54:50.504470 9941 fs_manager.cc:359] Metadata directory not provided
I20260925 04:54:50.504611 9941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:54:50.504963 9941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:54:50.508078 9941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-0/data/instance:
uuid: "0d49273afc8746468620f97237b9f0fb"
format_stamp: "Formatted at 2026-09-25 04:54:50 on dist-test-slave-xzjr"
server_key: "0883cc653683171483ffb436f44a9bd1"
server_key_iv: "1e9ea801cc7c2c0871524b0fcaa3d958"
server_key_version: "encryptionkey@0"
I20260925 04:54:50.508899 9941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance:
uuid: "0d49273afc8746468620f97237b9f0fb"
format_stamp: "Formatted at 2026-09-25 04:54:50 on dist-test-slave-xzjr"
server_key: "0883cc653683171483ffb436f44a9bd1"
server_key_iv: "1e9ea801cc7c2c0871524b0fcaa3d958"
server_key_version: "encryptionkey@0"
I20260925 04:54:50.514535 9941 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 04:54:50.518393 9956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:50.519876 9941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 04:54:50.520083 9941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-0/wal
uuid: "0d49273afc8746468620f97237b9f0fb"
format_stamp: "Formatted at 2026-09-25 04:54:50 on dist-test-slave-xzjr"
server_key: "0883cc653683171483ffb436f44a9bd1"
server_key_iv: "1e9ea801cc7c2c0871524b0fcaa3d958"
server_key_version: "encryptionkey@0"
I20260925 04:54:50.520316 9941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:54:50.537631 9941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:54:50.542306 9941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:54:50.542605 9941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:54:50.543604 9941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:54:50.545171 9941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:54:50.545264 9941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:50.545382 9941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:54:50.545451 9941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:50.579905 9941 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.129:35989
I20260925 04:54:50.579949 10068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.129:35989 every 8 connection(s)
I20260925 04:54:50.581528 9941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
I20260925 04:54:50.583160 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 9941
I20260925 04:54:50.583351 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance
I20260925 04:54:50.584123 6262 external_mini_cluster.cc:1428] Setting key 22a9e64f1ca93d3ea9d59e1cde60b1fb
Sep 25 04:54:50 dist-test-slave-xzjr krb5kdc[9835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312090, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.29.130@KRBTEST.COM; defaulting to no policy
I20260925 04:54:50.603971 10071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:50.584536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:48809 (local address 127.6.29.190:43343)
0925 04:54:50.585297 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:50.585303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:50.586346 (+ 1043us) server_negotiation.cc:407] Connection header received
0925 04:54:50.588352 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:50.588360 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:50.588452 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:50.588572 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:50.590101 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:50.591266 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:50.593374 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:50.593776 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:50.593889 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:50.598191 (+ 4302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:50.598196 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:50.598199 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:50.598232 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:50.600878 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:50.601632 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:50.601637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:50.601639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:50.601770 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:50.602219 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:50.602228 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:50.602230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:50.602495 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:50.602642 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:50.603363 (+ 721us) server_negotiation.cc:299] Negotiation successful
0925 04:54:50.603582 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":502,"threads_started":1}
I20260925 04:54:50.604974 10072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:50.585262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43343 (local address 127.6.29.129:48809)
0925 04:54:50.586184 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:50.586231 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:50.588088 (+ 1857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:50.588825 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:50.588834 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:50.589342 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:54:50.589924 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:50.589941 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:50.591456 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:50.591463 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:50.593229 (+ 1766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:50.593241 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:50.594024 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:50.594034 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:50.594391 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:50.595150 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:54:50.595186 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:54:50.597935 (+ 2749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:54:50.601037 (+ 3102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:50.601046 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:50.601066 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:50.601498 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:50.601886 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:50.601892 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:50.601895 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:50.602103 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:50.602621 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:54:50.602630 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:54:50.602994 (+ 364us) client_negotiation.cc:769] Sending connection context
0925 04:54:50.603251 (+ 257us) client_negotiation.cc:240] Negotiation successful
0925 04:54:50.603836 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":422,"threads_started":1}
I20260925 04:54:50.606669 10069 heartbeater.cc:347] Connected to a master server at 127.6.29.190:43343
I20260925 04:54:50.606953 10069 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.130@KRBTEST.COM" created.
I20260925 04:54:50.607852 10069 heartbeater.cc:511] Master 127.6.29.190:43343 requested a full tablet report, sending...
I20260925 04:54:50.610092 9880 ts_manager.cc:194] Registered new tserver with Master: 0d49273afc8746468620f97237b9f0fb (127.6.29.129:35989)
I20260925 04:54:50.611577 9880 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.129@KRBTEST.COM'} at 127.6.29.129:48809
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab.
Entry for principal kudu/127.6.29.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab.
I20260925 04:54:50.631472 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.130:0
--local_ip_for_outbound_sockets=127.6.29.130
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:43343
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:33785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:54:50.961710 10075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:54:50.962001 10075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:54:50.962158 10075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:54:50.969395 10075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:54:50.969511 10075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:54:50.969697 10075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.130
I20260925 04:54:50.979516 10075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:33785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:43343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.130
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:54:50.980885 10075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:54:50.982198 10075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:54:50.996868 10081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:50.997462 10080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:50.997484 10083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:54:50.997833 10075 server_base.cc:1034] running on GCE node
I20260925 04:54:50.998497 10075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:54:50.999583 10075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:54:51.000818 10075 hybrid_clock.cc:648] HybridClock initialized: now 1790312091000760 us; error 65 us; skew 500 ppm
I20260925 04:54:51.001148 10075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:54:51 dist-test-slave-xzjr krb5kdc[9835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312090, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:54:51.004401 10075 init.cc:377] Logged in from keytab as kudu/127.6.29.130@KRBTEST.COM (short username kudu)
I20260925 04:54:51.005877 10075 webserver.cc:460] Webserver started at http://127.6.29.130:40389/ using document root <none> and password file <none>
I20260925 04:54:51.006553 10075 fs_manager.cc:359] Metadata directory not provided
I20260925 04:54:51.006675 10075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:54:51.006934 10075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:54:51.010004 10075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-1/data/instance:
uuid: "e9ea13b12c3a41a1b4b93f0949c303ca"
format_stamp: "Formatted at 2026-09-25 04:54:51 on dist-test-slave-xzjr"
server_key: "b436c3f082f9962afd4a9c6832d58cfc"
server_key_iv: "7821ef49d90efd07b25444d05fb773b9"
server_key_version: "encryptionkey@0"
I20260925 04:54:51.010766 10075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance:
uuid: "e9ea13b12c3a41a1b4b93f0949c303ca"
format_stamp: "Formatted at 2026-09-25 04:54:51 on dist-test-slave-xzjr"
server_key: "b436c3f082f9962afd4a9c6832d58cfc"
server_key_iv: "7821ef49d90efd07b25444d05fb773b9"
server_key_version: "encryptionkey@0"
I20260925 04:54:51.016216 10075 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260925 04:54:51.020104 10090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:51.021535 10075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:54:51.021703 10075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-1/wal
uuid: "e9ea13b12c3a41a1b4b93f0949c303ca"
format_stamp: "Formatted at 2026-09-25 04:54:51 on dist-test-slave-xzjr"
server_key: "b436c3f082f9962afd4a9c6832d58cfc"
server_key_iv: "7821ef49d90efd07b25444d05fb773b9"
server_key_version: "encryptionkey@0"
I20260925 04:54:51.021970 10075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:54:51.041298 10075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:54:51.045913 10075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:54:51.046207 10075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:54:51.047186 10075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:54:51.048779 10075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:54:51.048854 10075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:51.048997 10075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:54:51.049048 10075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:51.084055 10075 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.130:44825
I20260925 04:54:51.084102 10202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.130:44825 every 8 connection(s)
I20260925 04:54:51.085390 10075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
I20260925 04:54:51.090993 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 10075
I20260925 04:54:51.091156 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance
I20260925 04:54:51.091588 6262 external_mini_cluster.cc:1428] Setting key 9e1ce9daa8d3bc00d760b64218ffa6d6
Sep 25 04:54:51 dist-test-slave-xzjr krb5kdc[9835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312090, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:54:51.107957 10071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:51.088195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:41291 (local address 127.6.29.190:43343)
0925 04:54:51.088409 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:51.088416 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:51.089854 (+ 1438us) server_negotiation.cc:407] Connection header received
0925 04:54:51.091793 (+ 1939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:51.091801 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:51.091893 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:51.092033 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:51.093962 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:51.095048 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:51.096830 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:51.097653 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:51.097814 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:51.101892 (+ 4078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:51.101897 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:51.101900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:51.101935 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:51.104647 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:51.105435 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:51.105445 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:51.105449 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:51.105570 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:51.106000 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:51.106007 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:51.106010 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:51.106338 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:51.106484 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:51.107485 (+ 1001us) server_negotiation.cc:299] Negotiation successful
0925 04:54:51.107656 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 04:54:51.109712 10205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:51.088768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43343 (local address 127.6.29.130:41291)
0925 04:54:51.089652 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:51.089703 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:51.091459 (+ 1756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:51.092332 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:51.092344 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:51.093012 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:54:51.093764 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:51.093788 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:51.095237 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:51.095241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:51.096680 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:51.096693 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:51.097887 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:51.097892 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:51.098108 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:51.098862 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:54:51.098891 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:54:51.101642 (+ 2751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:54:51.104808 (+ 3166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:51.104818 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:51.104838 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:51.105292 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:51.105680 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:51.105686 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:51.105689 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:51.105886 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:51.106501 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:54:51.106513 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:54:51.107044 (+ 531us) client_negotiation.cc:769] Sending connection context
0925 04:54:51.107438 (+ 394us) client_negotiation.cc:240] Negotiation successful
0925 04:54:51.108125 (+ 687us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":360,"threads_started":1}
WARNING: no policy specified for kudu/127.6.29.131@KRBTEST.COM; defaulting to no policy
I20260925 04:54:51.111294 10203 heartbeater.cc:347] Connected to a master server at 127.6.29.190:43343
I20260925 04:54:51.111582 10203 heartbeater.cc:464] Registering TS with master...
I20260925 04:54:51.112526 10203 heartbeater.cc:511] Master 127.6.29.190:43343 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.131@KRBTEST.COM" created.
I20260925 04:54:51.114074 9880 ts_manager.cc:194] Registered new tserver with Master: e9ea13b12c3a41a1b4b93f0949c303ca (127.6.29.130:44825)
I20260925 04:54:51.114895 9880 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.130@KRBTEST.COM'} at 127.6.29.130:41291
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab.
Entry for principal kudu/127.6.29.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab.
I20260925 04:54:51.139231 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.131:0
--local_ip_for_outbound_sockets=127.6.29.131
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:43343
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:33785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:54:51.467233 10208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:54:51.467568 10208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:54:51.467738 10208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:54:51.475349 10208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:54:51.475519 10208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:54:51.475706 10208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.131
I20260925 04:54:51.485838 10208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:33785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:43343
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.131
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:54:51.487251 10208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:54:51.488675 10208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:54:51.502348 10213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:51.502418 10214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:51.502385 10216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:54:51.503372 10208 server_base.cc:1034] running on GCE node
I20260925 04:54:51.504156 10208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:54:51.505223 10208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:54:51.506486 10208 hybrid_clock.cc:648] HybridClock initialized: now 1790312091506407 us; error 64 us; skew 500 ppm
I20260925 04:54:51.506793 10208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:54:51 dist-test-slave-xzjr krb5kdc[9835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312091, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:54:51.516103 10208 init.cc:377] Logged in from keytab as kudu/127.6.29.131@KRBTEST.COM (short username kudu)
I20260925 04:54:51.517752 10208 webserver.cc:460] Webserver started at http://127.6.29.131:41277/ using document root <none> and password file <none>
I20260925 04:54:51.518532 10208 fs_manager.cc:359] Metadata directory not provided
I20260925 04:54:51.518697 10208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:54:51.519019 10208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:54:51.522146 10208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-2/data/instance:
uuid: "4a1663c5848d469d943aa2d94a5a1892"
format_stamp: "Formatted at 2026-09-25 04:54:51 on dist-test-slave-xzjr"
server_key: "cba23c57f2e366af57edda0b612ce5c1"
server_key_iv: "e7acc1ff2d51f88c4e3d6b1ee0f8a51d"
server_key_version: "encryptionkey@0"
I20260925 04:54:51.522956 10208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance:
uuid: "4a1663c5848d469d943aa2d94a5a1892"
format_stamp: "Formatted at 2026-09-25 04:54:51 on dist-test-slave-xzjr"
server_key: "cba23c57f2e366af57edda0b612ce5c1"
server_key_iv: "e7acc1ff2d51f88c4e3d6b1ee0f8a51d"
server_key_version: "encryptionkey@0"
I20260925 04:54:51.529013 10208 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260925 04:54:51.533034 10223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:51.534579 10208 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 04:54:51.534786 10208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-2/wal
uuid: "4a1663c5848d469d943aa2d94a5a1892"
format_stamp: "Formatted at 2026-09-25 04:54:51 on dist-test-slave-xzjr"
server_key: "cba23c57f2e366af57edda0b612ce5c1"
server_key_iv: "e7acc1ff2d51f88c4e3d6b1ee0f8a51d"
server_key_version: "encryptionkey@0"
I20260925 04:54:51.535010 10208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:54:51.552431 10208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:54:51.556913 10208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:54:51.557222 10208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:54:51.558230 10208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:54:51.559732 10208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:54:51.559902 10208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260925 04:54:51.560056 10208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:54:51.560134 10208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:51.595343 10208 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.131:42123
I20260925 04:54:51.595366 10335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.131:42123 every 8 connection(s)
I20260925 04:54:51.596740 10208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
I20260925 04:54:51.599335 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 10208
I20260925 04:54:51.599519 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance
I20260925 04:54:51.600045 6262 external_mini_cluster.cc:1428] Setting key e188167dd8c94c857dc7f0214b06cfeb
Sep 25 04:54:51 dist-test-slave-xzjr krb5kdc[9835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312091, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:54:51.614741 10069 heartbeater.cc:503] Master 127.6.29.190:43343 was elected leader, sending a full tablet report...
I20260925 04:54:51.617820 10071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:51.599466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:43105 (local address 127.6.29.190:43343)
0925 04:54:51.599784 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:51.599790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:51.601088 (+ 1298us) server_negotiation.cc:407] Connection header received
0925 04:54:51.602931 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:51.602938 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:51.603055 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:51.603136 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:51.604697 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:51.605494 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:51.607133 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:51.607549 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:51.607654 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:51.611293 (+ 3639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:51.611298 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:51.611302 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:51.611343 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:51.614437 (+ 3094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:51.615373 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:51.615378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:51.615381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:51.615555 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:51.616086 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:51.616090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:51.616092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:51.616457 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:51.616567 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:51.617363 (+ 796us) server_negotiation.cc:299] Negotiation successful
0925 04:54:51.617527 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260925 04:54:51.618815 10338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:51.600046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43343 (local address 127.6.29.131:43105)
0925 04:54:51.600925 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:51.600972 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:51.602686 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:51.603367 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:51.603375 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:51.603953 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:54:51.604542 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:51.604560 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:51.605622 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:51.605626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:51.607003 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:51.607014 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:51.607832 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:51.607844 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:51.608058 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:51.608719 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:54:51.608741 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:54:51.611088 (+ 2347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:54:51.614604 (+ 3516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:51.614613 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:51.614632 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:51.615244 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:51.615685 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:51.615690 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:51.615693 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:51.615968 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:51.616588 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:54:51.616600 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:54:51.616966 (+ 366us) client_negotiation.cc:769] Sending connection context
0925 04:54:51.617251 (+ 285us) client_negotiation.cc:240] Negotiation successful
0925 04:54:51.617700 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":368,"threads_started":1}
I20260925 04:54:51.620321 10336 heartbeater.cc:347] Connected to a master server at 127.6.29.190:43343
I20260925 04:54:51.620596 10336 heartbeater.cc:464] Registering TS with master...
I20260925 04:54:51.621353 10336 heartbeater.cc:511] Master 127.6.29.190:43343 requested a full tablet report, sending...
I20260925 04:54:51.622712 9880 ts_manager.cc:194] Registered new tserver with Master: 4a1663c5848d469d943aa2d94a5a1892 (127.6.29.131:42123)
I20260925 04:54:51.623515 9880 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.131@KRBTEST.COM'} at 127.6.29.131:43105
I20260925 04:54:51.628224 6262 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 04:54:51 dist-test-slave-xzjr krb5kdc[9835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 04:54:51 dist-test-slave-xzjr krb5kdc[9835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.129@KRBTEST.COM
I20260925 04:54:51.678757 10343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:51.663031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42414 (local address 127.6.29.129:35989)
0925 04:54:51.663709 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:51.663721 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:51.663849 (+ 128us) server_negotiation.cc:407] Connection header received
0925 04:54:51.664069 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:51.664076 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:51.664300 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:51.664475 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:51.665812 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:51.666847 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:51.668191 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:51.668767 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:51.668913 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:51.673008 (+ 4095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:51.673021 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:51.673037 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:51.673079 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:51.676087 (+ 3008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:51.676751 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:51.676758 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:51.676761 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:51.676907 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:51.677267 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:51.677280 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:51.677282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:51.677692 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:51.677926 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:51.678227 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 04:54:51.678425 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":331,"threads_started":1}
Sep 25 04:54:51 dist-test-slave-xzjr krb5kdc[9835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:54:51.697299 10071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:51.683188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51822 (local address 127.6.29.190:43343)
0925 04:54:51.683392 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:51.683398 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:51.683526 (+ 128us) server_negotiation.cc:407] Connection header received
0925 04:54:51.683652 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:51.683656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:51.683720 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:51.683856 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:51.685051 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:51.685804 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:51.686990 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:51.687418 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:51.687527 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:51.691673 (+ 4146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:51.691679 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:51.691682 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:51.691720 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:51.694462 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:51.695160 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:51.695168 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:51.695173 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:51.695311 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:51.695682 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:51.695687 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:51.695690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:51.696240 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:51.696475 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:51.696778 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 04:54:51.696947 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260925 04:54:51.697943 9880 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51822
I20260925 04:54:51.699030 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 9941
I20260925 04:54:51.711831 10343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:51.700132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42428 (local address 127.6.29.129:35989)
0925 04:54:51.700333 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:51.700337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:51.700407 (+ 70us) server_negotiation.cc:407] Connection header received
0925 04:54:51.700538 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:51.700542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:51.700602 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:51.700681 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:51.701816 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:51.702497 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:51.703734 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:51.704209 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:51.704310 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:51.706557 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:51.706563 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:51.706566 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:51.706602 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:51.709276 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:51.709964 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:51.709967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:51.709970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:51.710083 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:51.710498 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:51.710502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:51.710504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:51.710769 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:51.710946 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:51.711203 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 04:54:51.711367 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 04:54:51.712471 10064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:54:51.796499 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 9941
I20260925 04:54:51.814831 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 10075
Sep 25 04:54:51 dist-test-slave-xzjr krb5kdc[9835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.130@KRBTEST.COM
I20260925 04:54:51.831240 10348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:51.815816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40076 (local address 127.6.29.130:44825)
0925 04:54:51.816304 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:51.816311 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:51.816336 (+ 25us) server_negotiation.cc:407] Connection header received
0925 04:54:51.816395 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:51.816401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:51.816616 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:51.816790 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:51.817930 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:51.818906 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:51.820255 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:51.820817 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:51.820958 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:51.825133 (+ 4175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:51.825145 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:51.825161 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:51.825197 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:51.828523 (+ 3326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:51.829270 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:51.829278 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:51.829281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:51.829401 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:51.829781 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:51.829784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:51.829787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:51.830109 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:51.830337 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:51.830688 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 04:54:51.830955 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":231,"threads_started":1}
I20260925 04:54:51.832711 10198 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:54:51.916430 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 10075
I20260925 04:54:51.933235 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 10208
Sep 25 04:54:51 dist-test-slave-xzjr krb5kdc[9835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.131@KRBTEST.COM
I20260925 04:54:51.950394 10350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:51.934195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46742 (local address 127.6.29.131:42123)
0925 04:54:51.934798 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:51.934805 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:51.934835 (+ 30us) server_negotiation.cc:407] Connection header received
0925 04:54:51.934894 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:51.934901 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:51.935174 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:51.935306 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:51.936900 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:51.937963 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:51.939438 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:51.940184 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:51.940294 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:51.944648 (+ 4354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:51.944667 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:51.944682 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:51.944722 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:51.947547 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:51.948367 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:51.948374 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:51.948376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:51.948496 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:51.948867 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:51.948870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:51.948872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:51.949204 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:51.949436 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:51.949784 (+ 348us) server_negotiation.cc:299] Negotiation successful
0925 04:54:51.949989 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":250,"threads_started":1}
I20260925 04:54:51.951833 10331 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:54:52.043309 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 10208
I20260925 04:54:52.059362 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 9848
I20260925 04:54:52.060629 9910 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:54:52.165723 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 9848
2026-09-25T04:54:52Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3901 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:54:52 dist-test-slave-xzjr krb5kdc[10355](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:54:52 dist-test-slave-xzjr krb5kdc[10355](info): set up 2 sockets
Sep 25 04:54:52 dist-test-slave-xzjr krb5kdc[10355](info): commencing operation
krb5kdc: starting...
W20260925 04:54:53.260995 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 04:54:53 dist-test-slave-xzjr krb5kdc[10355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:54:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:54:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.29.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
I20260925 04:54:53.463755 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:34095
--webserver_interface=127.6.29.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:44443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:34095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:54:53.796411 10368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:54:53.796754 10368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:54:53.796916 10368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:54:53.804436 10368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:54:53.804555 10368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:54:53.804649 10368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:54:53.804752 10368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:54:53.804829 10368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:54:53.815001 10368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:44443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:34095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:34095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:54:53.816390 10368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:54:53.817723 10368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:54:53.829708 10374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:53.829799 10373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:53.829818 10376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:54:53.830973 10368 server_base.cc:1034] running on GCE node
I20260925 04:54:53.831903 10368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:54:53.833426 10368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:54:53.834687 10368 hybrid_clock.cc:648] HybridClock initialized: now 1790312093834649 us; error 37 us; skew 500 ppm
I20260925 04:54:53.835072 10368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:54:53 dist-test-slave-xzjr krb5kdc[10355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312093, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:54:53.838450 10368 init.cc:377] Logged in from keytab as kudu/127.6.29.190@KRBTEST.COM (short username kudu)
I20260925 04:54:53.839851 10368 fs_manager.cc:359] Metadata directory not provided
I20260925 04:54:53.839983 10368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:54:53.840329 10368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:54:53.843487 10368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "79a1f9a2851e4c2d81586fc981aa5d53"
format_stamp: "Formatted at 2026-09-25 04:54:53 on dist-test-slave-xzjr"
server_key: "1468e1be79d4b07c672ee648b8f0488b"
server_key_iv: "8cd5a2043dff02e0681a889563f22c8e"
server_key_version: "encryptionkey@0"
I20260925 04:54:53.844352 10368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "79a1f9a2851e4c2d81586fc981aa5d53"
format_stamp: "Formatted at 2026-09-25 04:54:53 on dist-test-slave-xzjr"
server_key: "1468e1be79d4b07c672ee648b8f0488b"
server_key_iv: "8cd5a2043dff02e0681a889563f22c8e"
server_key_version: "encryptionkey@0"
I20260925 04:54:53.849498 10368 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 04:54:53.853022 10382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:53.854378 10368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:54:53.854566 10368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "79a1f9a2851e4c2d81586fc981aa5d53"
format_stamp: "Formatted at 2026-09-25 04:54:53 on dist-test-slave-xzjr"
server_key: "1468e1be79d4b07c672ee648b8f0488b"
server_key_iv: "8cd5a2043dff02e0681a889563f22c8e"
server_key_version: "encryptionkey@0"
I20260925 04:54:53.854797 10368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:54:53.880113 10368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:54:53.884320 10368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:54:53.884591 10368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:54:53.903841 10368 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:34095
I20260925 04:54:53.903873 10433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:34095 every 8 connection(s)
I20260925 04:54:53.904745 10368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:54:53.908569 10434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:54:53.914088 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 10368
I20260925 04:54:53.914314 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 04:54:53.914916 6262 external_mini_cluster.cc:1428] Setting key 3e42cb9453fe9a564d04cc6292da62a1
I20260925 04:54:53.919834 10434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79a1f9a2851e4c2d81586fc981aa5d53: Bootstrap starting.
I20260925 04:54:53.922956 10434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79a1f9a2851e4c2d81586fc981aa5d53: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:53.924093 10434 log.cc:824] T 00000000000000000000000000000000 P 79a1f9a2851e4c2d81586fc981aa5d53: Log is configured to *not* fsync() on all Append() calls
I20260925 04:54:53.927371 10434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79a1f9a2851e4c2d81586fc981aa5d53: No bootstrap required, opened a new log
Sep 25 04:54:53 dist-test-slave-xzjr krb5kdc[10355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:54:53.935708 10434 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79a1f9a2851e4c2d81586fc981aa5d53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79a1f9a2851e4c2d81586fc981aa5d53" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 34095 } }
I20260925 04:54:53.936177 10434 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79a1f9a2851e4c2d81586fc981aa5d53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:53.936285 10434 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79a1f9a2851e4c2d81586fc981aa5d53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79a1f9a2851e4c2d81586fc981aa5d53, State: Initialized, Role: FOLLOWER
I20260925 04:54:53.936944 10434 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79a1f9a2851e4c2d81586fc981aa5d53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79a1f9a2851e4c2d81586fc981aa5d53" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 34095 } }
I20260925 04:54:53.937201 10434 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79a1f9a2851e4c2d81586fc981aa5d53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:54:53.937351 10434 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79a1f9a2851e4c2d81586fc981aa5d53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:54:53.937568 10434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79a1f9a2851e4c2d81586fc981aa5d53 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:53.938563 10437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:53.917143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46150 (local address 127.6.29.190:34095)
0925 04:54:53.917949 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:53.917961 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:53.918011 (+ 50us) server_negotiation.cc:407] Connection header received
0925 04:54:53.919884 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:53.919909 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:53.920373 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:53.920796 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:53.922336 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:53.923737 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:53.925357 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:53.926026 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:53.926163 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:53.930840 (+ 4677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:53.930850 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:53.930872 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:53.930908 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:53.934122 (+ 3214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:53.934906 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:53.934915 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:53.934921 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:53.935080 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:53.935469 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:53.935473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:53.935475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:53.936134 (+ 659us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:53.936482 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:53.936901 (+ 419us) server_negotiation.cc:299] Negotiation successful
0925 04:54:53.937406 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":336,"threads_started":1}
I20260925 04:54:53.940217 10434 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79a1f9a2851e4c2d81586fc981aa5d53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79a1f9a2851e4c2d81586fc981aa5d53" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 34095 } }
I20260925 04:54:53.940824 10434 leader_election.cc:302] T 00000000000000000000000000000000 P 79a1f9a2851e4c2d81586fc981aa5d53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79a1f9a2851e4c2d81586fc981aa5d53; no voters:
I20260925 04:54:53.941490 10434 leader_election.cc:288] T 00000000000000000000000000000000 P 79a1f9a2851e4c2d81586fc981aa5d53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:54:53.941635 10439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79a1f9a2851e4c2d81586fc981aa5d53 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:54:53.942359 10439 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79a1f9a2851e4c2d81586fc981aa5d53 [term 1 LEADER]: Becoming Leader. State: Replica: 79a1f9a2851e4c2d81586fc981aa5d53, State: Running, Role: LEADER
I20260925 04:54:53.942895 10439 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79a1f9a2851e4c2d81586fc981aa5d53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79a1f9a2851e4c2d81586fc981aa5d53" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 34095 } }
I20260925 04:54:53.943946 10434 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79a1f9a2851e4c2d81586fc981aa5d53 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:54:53.945778 10441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79a1f9a2851e4c2d81586fc981aa5d53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79a1f9a2851e4c2d81586fc981aa5d53. Latest consensus state: current_term: 1 leader_uuid: "79a1f9a2851e4c2d81586fc981aa5d53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79a1f9a2851e4c2d81586fc981aa5d53" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 34095 } } }
I20260925 04:54:53.945779 10440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79a1f9a2851e4c2d81586fc981aa5d53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79a1f9a2851e4c2d81586fc981aa5d53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79a1f9a2851e4c2d81586fc981aa5d53" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 34095 } } }
I20260925 04:54:53.946380 10440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79a1f9a2851e4c2d81586fc981aa5d53 [sys.catalog]: This master's current role is: LEADER
I20260925 04:54:53.946388 10441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79a1f9a2851e4c2d81586fc981aa5d53 [sys.catalog]: This master's current role is: LEADER
I20260925 04:54:53.949879 10445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:54:53.955598 10445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:54:53.964483 10445 catalog_manager.cc:1347] Generated new cluster ID: 23eeeca9df97488aa9bd8b6405f26ac9
I20260925 04:54:53.964613 10445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:54:54.013342 10445 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:54:54.014673 10445 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:54:54.024658 10445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79a1f9a2851e4c2d81586fc981aa5d53: Generated new TSK 0
I20260925 04:54:54.025596 10445 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.29.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab.
Entry for principal kudu/127.6.29.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab.
I20260925 04:54:54.088356 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.129:0
--local_ip_for_outbound_sockets=127.6.29.129
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:34095
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:44443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:54:54.430670 10460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:54:54.431000 10460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:54:54.431164 10460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:54:54.438534 10460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:54:54.438710 10460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:54:54.438881 10460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.129
I20260925 04:54:54.449098 10460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:44443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:34095
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.129
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:54:54.450495 10460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:54:54.451910 10460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:54:54.465566 10465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:54.465711 10466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:54.465693 10468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:54:54.466398 10460 server_base.cc:1034] running on GCE node
I20260925 04:54:54.466976 10460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:54:54.467984 10460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:54:54.469230 10460 hybrid_clock.cc:648] HybridClock initialized: now 1790312094469159 us; error 61 us; skew 500 ppm
I20260925 04:54:54.469578 10460 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:54:54 dist-test-slave-xzjr krb5kdc[10355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312094, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:54:54.472841 10460 init.cc:377] Logged in from keytab as kudu/127.6.29.129@KRBTEST.COM (short username kudu)
I20260925 04:54:54.474011 10460 fs_manager.cc:359] Metadata directory not provided
I20260925 04:54:54.474143 10460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:54:54.474581 10460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:54:54.477631 10460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-0/data/instance:
uuid: "14e3e6ef72544f309b3fc3d9d59b6aa7"
format_stamp: "Formatted at 2026-09-25 04:54:54 on dist-test-slave-xzjr"
server_key: "d61d75f10f3ba3c1cfb8ef7aa5bff3f6"
server_key_iv: "33de39ffce6bf38e3798b04f8468a63d"
server_key_version: "encryptionkey@0"
I20260925 04:54:54.478417 10460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance:
uuid: "14e3e6ef72544f309b3fc3d9d59b6aa7"
format_stamp: "Formatted at 2026-09-25 04:54:54 on dist-test-slave-xzjr"
server_key: "d61d75f10f3ba3c1cfb8ef7aa5bff3f6"
server_key_iv: "33de39ffce6bf38e3798b04f8468a63d"
server_key_version: "encryptionkey@0"
I20260925 04:54:54.483603 10460 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 04:54:54.487267 10474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:54.488806 10460 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:54:54.488999 10460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-0/wal
uuid: "14e3e6ef72544f309b3fc3d9d59b6aa7"
format_stamp: "Formatted at 2026-09-25 04:54:54 on dist-test-slave-xzjr"
server_key: "d61d75f10f3ba3c1cfb8ef7aa5bff3f6"
server_key_iv: "33de39ffce6bf38e3798b04f8468a63d"
server_key_version: "encryptionkey@0"
I20260925 04:54:54.489224 10460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:54:54.510895 10460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:54:54.515396 10460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:54:54.515739 10460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:54:54.516786 10460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:54:54.518225 10460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:54:54.518348 10460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:54.518468 10460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:54:54.518551 10460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:54.552415 10460 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.129:44315
I20260925 04:54:54.552472 10586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.129:44315 every 8 connection(s)
I20260925 04:54:54.553526 10460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
I20260925 04:54:54.560470 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 10460
I20260925 04:54:54.560701 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance
I20260925 04:54:54.561326 6262 external_mini_cluster.cc:1428] Setting key fc375fdb251189ebe592c5508f95d9dc
Sep 25 04:54:54 dist-test-slave-xzjr krb5kdc[10355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312094, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:54:54.574455 10589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:54.556534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:48071 (local address 127.6.29.190:34095)
0925 04:54:54.556993 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:54.556997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:54.557944 (+ 947us) server_negotiation.cc:407] Connection header received
0925 04:54:54.559869 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:54.559875 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:54.559967 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:54.560081 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:54.561778 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:54.562611 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:54.564399 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:54.564817 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:54.564956 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:54.568780 (+ 3824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:54.568786 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:54.568789 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:54.568821 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:54.571379 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:54.572189 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:54.572195 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:54.572198 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:54.572310 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:54.572738 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:54.572747 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:54.572749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:54.573001 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:54.573151 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:54.573869 (+ 718us) server_negotiation.cc:299] Negotiation successful
0925 04:54:54.574091 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":258,"threads_started":1}
I20260925 04:54:54.575500 10590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:54.557005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:34095 (local address 127.6.29.129:48071)
0925 04:54:54.557777 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:54.557817 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:54.559574 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:54.560320 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:54.560328 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:54.560965 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:54:54.561610 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:54.561630 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:54.562770 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:54.562775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:54.564242 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:54.564255 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:54.565050 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:54.565057 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:54.565300 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:54.566075 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:54:54.566099 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:54:54.568512 (+ 2413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:54:54.571532 (+ 3020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:54.571542 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:54.571559 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:54.572060 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:54.572420 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:54.572425 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:54.572428 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:54.572613 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:54.573133 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:54:54.573140 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:54:54.573487 (+ 347us) client_negotiation.cc:769] Sending connection context
0925 04:54:54.573757 (+ 270us) client_negotiation.cc:240] Negotiation successful
0925 04:54:54.574326 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":313,"threads_started":1}
I20260925 04:54:54.577389 10587 heartbeater.cc:347] Connected to a master server at 127.6.29.190:34095
I20260925 04:54:54.577728 10587 heartbeater.cc:464] Registering TS with master...
I20260925 04:54:54.578402 10587 heartbeater.cc:511] Master 127.6.29.190:34095 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.29.130@KRBTEST.COM; defaulting to no policy
I20260925 04:54:54.580721 10399 ts_manager.cc:194] Registered new tserver with Master: 14e3e6ef72544f309b3fc3d9d59b6aa7 (127.6.29.129:44315)
I20260925 04:54:54.582229 10399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.129@KRBTEST.COM'} at 127.6.29.129:48071
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab.
Entry for principal kudu/127.6.29.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab.
I20260925 04:54:54.608031 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.130:0
--local_ip_for_outbound_sockets=127.6.29.130
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:34095
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:44443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:54:54.934135 10593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:54:54.934446 10593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:54:54.934625 10593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:54:54.942092 10593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:54:54.942222 10593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:54:54.942421 10593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.130
I20260925 04:54:54.952668 10593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:44443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:34095
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.130
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:54:54.954080 10593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:54:54.955449 10593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:54:54.969400 10598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:54.969324 10599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:54.969376 10601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:54:54.970193 10593 server_base.cc:1034] running on GCE node
I20260925 04:54:54.970808 10593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:54:54.971755 10593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:54:54.973045 10593 hybrid_clock.cc:648] HybridClock initialized: now 1790312094972985 us; error 56 us; skew 500 ppm
I20260925 04:54:54.973378 10593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:54:54 dist-test-slave-xzjr krb5kdc[10355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312094, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:54:54.976616 10593 init.cc:377] Logged in from keytab as kudu/127.6.29.130@KRBTEST.COM (short username kudu)
I20260925 04:54:54.977941 10593 fs_manager.cc:359] Metadata directory not provided
I20260925 04:54:54.978075 10593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:54:54.978392 10593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:54:54.981473 10593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-1/data/instance:
uuid: "fd486120971d4be18bf0a84c60dd6394"
format_stamp: "Formatted at 2026-09-25 04:54:54 on dist-test-slave-xzjr"
server_key: "c0d90ce85c748f9b8d2628c24de77b68"
server_key_iv: "69485c41dfe6d470a30080a788339434"
server_key_version: "encryptionkey@0"
I20260925 04:54:54.982324 10593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance:
uuid: "fd486120971d4be18bf0a84c60dd6394"
format_stamp: "Formatted at 2026-09-25 04:54:54 on dist-test-slave-xzjr"
server_key: "c0d90ce85c748f9b8d2628c24de77b68"
server_key_iv: "69485c41dfe6d470a30080a788339434"
server_key_version: "encryptionkey@0"
I20260925 04:54:54.987640 10593 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 04:54:54.991472 10607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:54.993125 10593 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:54:54.993323 10593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-1/wal
uuid: "fd486120971d4be18bf0a84c60dd6394"
format_stamp: "Formatted at 2026-09-25 04:54:54 on dist-test-slave-xzjr"
server_key: "c0d90ce85c748f9b8d2628c24de77b68"
server_key_iv: "69485c41dfe6d470a30080a788339434"
server_key_version: "encryptionkey@0"
I20260925 04:54:54.993561 10593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:54:55.007752 10593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:54:55.012239 10593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:54:55.012523 10593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:54:55.013523 10593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:54:55.014925 10593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:54:55.015049 10593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:55.015182 10593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:54:55.015271 10593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:55.048241 10593 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.130:39157
I20260925 04:54:55.048296 10719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.130:39157 every 8 connection(s)
I20260925 04:54:55.049139 10593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
I20260925 04:54:55.057431 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 10593
I20260925 04:54:55.057613 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance
I20260925 04:54:55.058133 6262 external_mini_cluster.cc:1428] Setting key eaf326c2765ea5b1a70c02e867cd5142
Sep 25 04:54:55 dist-test-slave-xzjr krb5kdc[10355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312094, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:54:55.072405 10589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:55.051747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:53823 (local address 127.6.29.190:34095)
0925 04:54:55.051956 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:55.051960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:55.053197 (+ 1237us) server_negotiation.cc:407] Connection header received
0925 04:54:55.055132 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:55.055137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:55.055213 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:55.055298 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:55.057081 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:55.058199 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:55.059967 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:55.060573 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:55.060705 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:55.064900 (+ 4195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:55.064906 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:55.064910 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:55.064945 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:55.068924 (+ 3979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:55.069676 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:55.069685 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:55.069702 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:55.069868 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:55.070285 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:55.070294 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:55.070301 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:55.070737 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:55.070895 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:55.071594 (+ 699us) server_negotiation.cc:299] Negotiation successful
0925 04:54:55.071905 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 04:54:55.073055 10722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:55.052226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:34095 (local address 127.6.29.130:53823)
0925 04:54:55.052999 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:55.053045 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:55.054836 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:55.055564 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:55.055574 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:55.056247 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:54:55.056857 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:55.056875 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:55.058334 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:55.058338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:55.059819 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:55.059835 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:55.060800 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:55.060808 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:55.061039 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:55.061802 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:54:55.061831 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:54:55.064580 (+ 2749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:54:55.069079 (+ 4499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:55.069086 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:55.069101 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:55.069529 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:55.069982 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:55.069985 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:55.069988 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:55.070167 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:55.070868 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:54:55.070873 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:54:55.071198 (+ 325us) client_negotiation.cc:769] Sending connection context
0925 04:54:55.071425 (+ 227us) client_negotiation.cc:240] Negotiation successful
0925 04:54:55.071874 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":321,"threads_started":1}
I20260925 04:54:55.074612 10720 heartbeater.cc:347] Connected to a master server at 127.6.29.190:34095
I20260925 04:54:55.074978 10720 heartbeater.cc:464] Registering TS with master...
I20260925 04:54:55.075680 10720 heartbeater.cc:511] Master 127.6.29.190:34095 requested a full tablet report, sending...
I20260925 04:54:55.077414 10399 ts_manager.cc:194] Registered new tserver with Master: fd486120971d4be18bf0a84c60dd6394 (127.6.29.130:39157)
I20260925 04:54:55.078215 10399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.130@KRBTEST.COM'} at 127.6.29.130:53823
WARNING: no policy specified for kudu/127.6.29.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab.
Entry for principal kudu/127.6.29.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab.
I20260925 04:54:55.108930 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.131:0
--local_ip_for_outbound_sockets=127.6.29.131
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:34095
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:44443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:54:55.446846 10725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:54:55.447223 10725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:54:55.447337 10725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:54:55.454854 10725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:54:55.454969 10725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:54:55.455087 10725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.131
I20260925 04:54:55.465301 10725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:44443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:34095
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.131
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:54:55.466614 10725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:54:55.468032 10725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:54:55.480883 10730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:55.480923 10733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:55.480973 10731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:54:55.482208 10725 server_base.cc:1034] running on GCE node
I20260925 04:54:55.482810 10725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:54:55.483892 10725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:54:55.485122 10725 hybrid_clock.cc:648] HybridClock initialized: now 1790312095485069 us; error 54 us; skew 500 ppm
I20260925 04:54:55.485469 10725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:54:55 dist-test-slave-xzjr krb5kdc[10355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312095, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:54:55.488642 10725 init.cc:377] Logged in from keytab as kudu/127.6.29.131@KRBTEST.COM (short username kudu)
I20260925 04:54:55.489898 10725 fs_manager.cc:359] Metadata directory not provided
I20260925 04:54:55.490029 10725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:54:55.490345 10725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:54:55.493454 10725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-2/data/instance:
uuid: "e65d7ed29def474494a2f6fb87074c06"
format_stamp: "Formatted at 2026-09-25 04:54:55 on dist-test-slave-xzjr"
server_key: "f66dfb1f7e71cf033cda6a90a840a526"
server_key_iv: "2b93fa81c6512ce6a7c498f74a8e1eea"
server_key_version: "encryptionkey@0"
I20260925 04:54:55.494331 10725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance:
uuid: "e65d7ed29def474494a2f6fb87074c06"
format_stamp: "Formatted at 2026-09-25 04:54:55 on dist-test-slave-xzjr"
server_key: "f66dfb1f7e71cf033cda6a90a840a526"
server_key_iv: "2b93fa81c6512ce6a7c498f74a8e1eea"
server_key_version: "encryptionkey@0"
I20260925 04:54:55.499891 10725 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 04:54:55.503429 10739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:55.505136 10725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:54:55.505336 10725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-2/wal
uuid: "e65d7ed29def474494a2f6fb87074c06"
format_stamp: "Formatted at 2026-09-25 04:54:55 on dist-test-slave-xzjr"
server_key: "f66dfb1f7e71cf033cda6a90a840a526"
server_key_iv: "2b93fa81c6512ce6a7c498f74a8e1eea"
server_key_version: "encryptionkey@0"
I20260925 04:54:55.505553 10725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:54:55.540632 10725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:54:55.545113 10725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:54:55.545394 10725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:54:55.546422 10725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:54:55.547755 10725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:54:55.547868 10725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:55.548015 10725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:54:55.548076 10725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:55.583241 10725 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.131:46111
I20260925 04:54:55.583292 10851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.131:46111 every 8 connection(s)
I20260925 04:54:55.584200 10725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
I20260925 04:54:55.585423 10587 heartbeater.cc:503] Master 127.6.29.190:34095 was elected leader, sending a full tablet report...
I20260925 04:54:55.590152 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 10725
I20260925 04:54:55.590327 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance
I20260925 04:54:55.590751 6262 external_mini_cluster.cc:1428] Setting key dc47d135545be52916f040ba826a8f0c
Sep 25 04:54:55 dist-test-slave-xzjr krb5kdc[10355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312095, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:54:55.604599 10854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:55.586660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:39807 (local address 127.6.29.190:34095)
0925 04:54:55.587181 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:55.587187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:55.588210 (+ 1023us) server_negotiation.cc:407] Connection header received
0925 04:54:55.589965 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:55.589974 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:55.590072 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:55.590211 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:55.591921 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:55.593015 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:55.594759 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:55.595171 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:55.595267 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:55.598889 (+ 3622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:55.598894 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:55.598897 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:55.598934 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:55.601541 (+ 2607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:55.602287 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:55.602291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:55.602294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:55.602416 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:55.602793 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:55.602796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:55.602798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:55.603062 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:55.603193 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:55.603971 (+ 778us) server_negotiation.cc:299] Negotiation successful
0925 04:54:55.604224 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":246,"threads_started":1}
I20260925 04:54:55.605654 10855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:55.587309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:34095 (local address 127.6.29.131:39807)
0925 04:54:55.588055 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:55.588092 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:55.589660 (+ 1568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:55.590427 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:55.590438 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:55.591024 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:54:55.591713 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:55.591731 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:55.593154 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:55.593159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:55.594616 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:55.594628 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:55.595353 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:55.595357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:55.595537 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:55.596260 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:54:55.596282 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:54:55.598674 (+ 2392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:54:55.601682 (+ 3008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:55.601691 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:55.601710 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:55.602145 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:55.602508 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:54:55.602511 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:54:55.602513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:54:55.602689 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:54:55.603181 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:54:55.603198 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:54:55.603520 (+ 322us) client_negotiation.cc:769] Sending connection context
0925 04:54:55.603820 (+ 300us) client_negotiation.cc:240] Negotiation successful
0925 04:54:55.604298 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":287,"threads_started":1}
I20260925 04:54:55.606951 10852 heartbeater.cc:347] Connected to a master server at 127.6.29.190:34095
I20260925 04:54:55.607240 10852 heartbeater.cc:464] Registering TS with master...
I20260925 04:54:55.607983 10852 heartbeater.cc:511] Master 127.6.29.190:34095 requested a full tablet report, sending...
I20260925 04:54:55.609360 10399 ts_manager.cc:194] Registered new tserver with Master: e65d7ed29def474494a2f6fb87074c06 (127.6.29.131:46111)
I20260925 04:54:55.610270 10399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.131@KRBTEST.COM'} at 127.6.29.131:39807
I20260925 04:54:55.618721 6262 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:54:55.636480 10854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:55.623587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46154 (local address 127.6.29.190:34095)
0925 04:54:55.623832 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:55.623838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:55.624306 (+ 468us) server_negotiation.cc:407] Connection header received
0925 04:54:55.624455 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:55.624459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:55.624526 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:55.624615 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:55.626087 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:55.626787 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:55.628338 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:55.628803 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:55.628908 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:55.631223 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:55.631228 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:55.631231 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:55.631266 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:55.633956 (+ 2690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:55.634709 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:55.634714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:55.634718 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:55.634850 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:55.635173 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:55.635176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:55.635178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:55.635490 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:55.635670 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:55.636053 (+ 383us) server_negotiation.cc:299] Negotiation successful
0925 04:54:55.636252 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 04:54:55.640727 10399 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:54:55.647651 10399 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:54:55.671872 10865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:55.660376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:39157 (local address 127.0.0.1:54172)
0925 04:54:55.661213 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:55.661255 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:55.661813 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:55.663811 (+ 1998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:55.663826 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:55.663870 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:54:55.664335 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:55.664348 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:55.666812 (+ 2464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:55.666817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:55.669231 (+ 2414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:55.669245 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:55.670918 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:55.670923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:55.671059 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:55.671112 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 04:54:55.671238 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 04:54:55.671378 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"mutex_wait_us":166,"thread_start_us":457,"threads_started":1}
I20260925 04:54:55.671867 10867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:55.660536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54172 (local address 127.6.29.130:39157)
0925 04:54:55.663060 (+ 2524us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:55.663073 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:55.663105 (+ 32us) server_negotiation.cc:407] Connection header received
0925 04:54:55.663231 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:55.663242 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:55.663524 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:55.664576 (+ 1052us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:54:55.664817 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:55.666652 (+ 1835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:55.669425 (+ 2773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:55.670711 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:55.670849 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:55.671052 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:55.671292 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 04:54:55.671462 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2330,"thread_start_us":2202,"threads_started":1}
I20260925 04:54:55.672502 10866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:55.660551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:44315 (local address 127.0.0.1:50194)
0925 04:54:55.662391 (+ 1840us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:55.662413 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:55.662531 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:55.664373 (+ 1842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:55.664377 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:55.664403 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:54:55.664804 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:55.664814 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:55.666855 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:55.666861 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:55.669981 (+ 3120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:55.669996 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:55.671832 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:55.671841 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:55.672031 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:55.672066 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 04:54:55.672151 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 04:54:55.672242 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1718,"mutex_wait_us":285,"thread_start_us":1568,"threads_started":1}
I20260925 04:54:55.674891 10868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:55.661051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50194 (local address 127.6.29.129:44315)
0925 04:54:55.663454 (+ 2403us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:55.663465 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:55.663494 (+ 29us) server_negotiation.cc:407] Connection header received
0925 04:54:55.663594 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:55.663600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:55.663890 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:55.664106 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:54:55.664978 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:55.666681 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:55.670176 (+ 3495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:55.671558 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:55.671705 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:55.672116 (+ 411us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:55.672211 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 04:54:55.672380 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2219,"thread_start_us":2066,"threads_started":1}
I20260925 04:54:55.676395 10655 tablet_service.cc:1467] Processing CreateTablet for tablet 134a9e65efd248c7b344a3c051a9dec6 (DEFAULT_TABLE table=test-table [id=7943d7f11a2a4690b75afd05da1f8d32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:54:55.677707 10870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:55.662238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47934 (local address 127.6.29.131:46111)
0925 04:54:55.663739 (+ 1501us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:55.663746 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:55.664347 (+ 601us) server_negotiation.cc:407] Connection header received
0925 04:54:55.664536 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:55.664541 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:55.664693 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:55.664873 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:54:55.665506 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:55.666704 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:55.672938 (+ 6234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:55.674392 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:55.674560 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:55.674811 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:55.677190 (+ 2379us) server_negotiation.cc:299] Negotiation successful
0925 04:54:55.677374 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1341,"thread_start_us":1207,"threads_started":1}
I20260925 04:54:55.677809 10655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 134a9e65efd248c7b344a3c051a9dec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:54:55.678068 10869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:55.661612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.131:46111 (local address 127.0.0.1:47934)
0925 04:54:55.664219 (+ 2607us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:55.664242 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:55.664372 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:55.664865 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:55.664869 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:55.664892 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:54:55.665291 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:55.665303 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:55.666857 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:55.666861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:55.669981 (+ 3120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:55.669996 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:55.674654 (+ 4658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:55.674664 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:55.674867 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:55.674908 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 04:54:55.677111 (+ 2203us) client_negotiation.cc:240] Negotiation successful
0925 04:54:55.677644 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2467,"mutex_wait_us":269,"thread_start_us":2297,"threads_started":1}
I20260925 04:54:55.678323 10522 tablet_service.cc:1467] Processing CreateTablet for tablet 134a9e65efd248c7b344a3c051a9dec6 (DEFAULT_TABLE table=test-table [id=7943d7f11a2a4690b75afd05da1f8d32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:54:55.679746 10522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 134a9e65efd248c7b344a3c051a9dec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:54:55.681838 10787 tablet_service.cc:1467] Processing CreateTablet for tablet 134a9e65efd248c7b344a3c051a9dec6 (DEFAULT_TABLE table=test-table [id=7943d7f11a2a4690b75afd05da1f8d32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:54:55.683269 10787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 134a9e65efd248c7b344a3c051a9dec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:54:55.690652 10871 tablet_bootstrap.cc:492] T 134a9e65efd248c7b344a3c051a9dec6 P 14e3e6ef72544f309b3fc3d9d59b6aa7: Bootstrap starting.
I20260925 04:54:55.694839 10871 tablet_bootstrap.cc:654] T 134a9e65efd248c7b344a3c051a9dec6 P 14e3e6ef72544f309b3fc3d9d59b6aa7: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:55.695950 10871 log.cc:824] T 134a9e65efd248c7b344a3c051a9dec6 P 14e3e6ef72544f309b3fc3d9d59b6aa7: Log is configured to *not* fsync() on all Append() calls
I20260925 04:54:55.698969 10871 tablet_bootstrap.cc:492] T 134a9e65efd248c7b344a3c051a9dec6 P 14e3e6ef72544f309b3fc3d9d59b6aa7: No bootstrap required, opened a new log
I20260925 04:54:55.699057 10874 tablet_bootstrap.cc:492] T 134a9e65efd248c7b344a3c051a9dec6 P e65d7ed29def474494a2f6fb87074c06: Bootstrap starting.
I20260925 04:54:55.699251 10871 ts_tablet_manager.cc:1397] T 134a9e65efd248c7b344a3c051a9dec6 P 14e3e6ef72544f309b3fc3d9d59b6aa7: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 04:54:55.702368 10874 tablet_bootstrap.cc:654] T 134a9e65efd248c7b344a3c051a9dec6 P e65d7ed29def474494a2f6fb87074c06: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:55.703365 10874 log.cc:824] T 134a9e65efd248c7b344a3c051a9dec6 P e65d7ed29def474494a2f6fb87074c06: Log is configured to *not* fsync() on all Append() calls
I20260925 04:54:55.704008 10872 tablet_bootstrap.cc:492] T 134a9e65efd248c7b344a3c051a9dec6 P fd486120971d4be18bf0a84c60dd6394: Bootstrap starting.
I20260925 04:54:55.706663 10874 tablet_bootstrap.cc:492] T 134a9e65efd248c7b344a3c051a9dec6 P e65d7ed29def474494a2f6fb87074c06: No bootstrap required, opened a new log
I20260925 04:54:55.706959 10874 ts_tablet_manager.cc:1397] T 134a9e65efd248c7b344a3c051a9dec6 P e65d7ed29def474494a2f6fb87074c06: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260925 04:54:55.707150 10871 raft_consensus.cc:348] T 134a9e65efd248c7b344a3c051a9dec6 P 14e3e6ef72544f309b3fc3d9d59b6aa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd486120971d4be18bf0a84c60dd6394" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 39157 } } peers { permanent_uuid: "14e3e6ef72544f309b3fc3d9d59b6aa7" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 44315 } } peers { permanent_uuid: "e65d7ed29def474494a2f6fb87074c06" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 46111 } }
I20260925 04:54:55.707535 10871 raft_consensus.cc:374] T 134a9e65efd248c7b344a3c051a9dec6 P 14e3e6ef72544f309b3fc3d9d59b6aa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:55.707653 10872 tablet_bootstrap.cc:654] T 134a9e65efd248c7b344a3c051a9dec6 P fd486120971d4be18bf0a84c60dd6394: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:55.707662 10871 raft_consensus.cc:729] T 134a9e65efd248c7b344a3c051a9dec6 P 14e3e6ef72544f309b3fc3d9d59b6aa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14e3e6ef72544f309b3fc3d9d59b6aa7, State: Initialized, Role: FOLLOWER
I20260925 04:54:55.708320 10871 consensus_queue.cc:260] T 134a9e65efd248c7b344a3c051a9dec6 P 14e3e6ef72544f309b3fc3d9d59b6aa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd486120971d4be18bf0a84c60dd6394" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 39157 } } peers { permanent_uuid: "14e3e6ef72544f309b3fc3d9d59b6aa7" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 44315 } } peers { permanent_uuid: "e65d7ed29def474494a2f6fb87074c06" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 46111 } }
I20260925 04:54:55.708986 10872 log.cc:824] T 134a9e65efd248c7b344a3c051a9dec6 P fd486120971d4be18bf0a84c60dd6394: Log is configured to *not* fsync() on all Append() calls
I20260925 04:54:55.710124 10871 ts_tablet_manager.cc:1428] T 134a9e65efd248c7b344a3c051a9dec6 P 14e3e6ef72544f309b3fc3d9d59b6aa7: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260925 04:54:55.713553 10872 tablet_bootstrap.cc:492] T 134a9e65efd248c7b344a3c051a9dec6 P fd486120971d4be18bf0a84c60dd6394: No bootstrap required, opened a new log
I20260925 04:54:55.713850 10872 ts_tablet_manager.cc:1397] T 134a9e65efd248c7b344a3c051a9dec6 P fd486120971d4be18bf0a84c60dd6394: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.000s
I20260925 04:54:55.718142 10874 raft_consensus.cc:348] T 134a9e65efd248c7b344a3c051a9dec6 P e65d7ed29def474494a2f6fb87074c06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd486120971d4be18bf0a84c60dd6394" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 39157 } } peers { permanent_uuid: "14e3e6ef72544f309b3fc3d9d59b6aa7" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 44315 } } peers { permanent_uuid: "e65d7ed29def474494a2f6fb87074c06" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 46111 } }
I20260925 04:54:55.718621 10874 raft_consensus.cc:374] T 134a9e65efd248c7b344a3c051a9dec6 P e65d7ed29def474494a2f6fb87074c06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:55.718734 10874 raft_consensus.cc:729] T 134a9e65efd248c7b344a3c051a9dec6 P e65d7ed29def474494a2f6fb87074c06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e65d7ed29def474494a2f6fb87074c06, State: Initialized, Role: FOLLOWER
I20260925 04:54:55.719429 10874 consensus_queue.cc:260] T 134a9e65efd248c7b344a3c051a9dec6 P e65d7ed29def474494a2f6fb87074c06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd486120971d4be18bf0a84c60dd6394" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 39157 } } peers { permanent_uuid: "14e3e6ef72544f309b3fc3d9d59b6aa7" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 44315 } } peers { permanent_uuid: "e65d7ed29def474494a2f6fb87074c06" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 46111 } }
I20260925 04:54:55.720609 10852 heartbeater.cc:503] Master 127.6.29.190:34095 was elected leader, sending a full tablet report...
I20260925 04:54:55.721372 10874 ts_tablet_manager.cc:1428] T 134a9e65efd248c7b344a3c051a9dec6 P e65d7ed29def474494a2f6fb87074c06: Time spent starting tablet: real 0.014s user 0.015s sys 0.001s
I20260925 04:54:55.723691 10872 raft_consensus.cc:348] T 134a9e65efd248c7b344a3c051a9dec6 P fd486120971d4be18bf0a84c60dd6394 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd486120971d4be18bf0a84c60dd6394" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 39157 } } peers { permanent_uuid: "14e3e6ef72544f309b3fc3d9d59b6aa7" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 44315 } } peers { permanent_uuid: "e65d7ed29def474494a2f6fb87074c06" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 46111 } }
I20260925 04:54:55.724171 10872 raft_consensus.cc:374] T 134a9e65efd248c7b344a3c051a9dec6 P fd486120971d4be18bf0a84c60dd6394 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:55.724287 10872 raft_consensus.cc:729] T 134a9e65efd248c7b344a3c051a9dec6 P fd486120971d4be18bf0a84c60dd6394 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd486120971d4be18bf0a84c60dd6394, State: Initialized, Role: FOLLOWER
I20260925 04:54:55.724889 10872 consensus_queue.cc:260] T 134a9e65efd248c7b344a3c051a9dec6 P fd486120971d4be18bf0a84c60dd6394 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd486120971d4be18bf0a84c60dd6394" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 39157 } } peers { permanent_uuid: "14e3e6ef72544f309b3fc3d9d59b6aa7" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 44315 } } peers { permanent_uuid: "e65d7ed29def474494a2f6fb87074c06" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 46111 } }
I20260925 04:54:55.726095 10720 heartbeater.cc:503] Master 127.6.29.190:34095 was elected leader, sending a full tablet report...
I20260925 04:54:55.727837 10872 ts_tablet_manager.cc:1428] T 134a9e65efd248c7b344a3c051a9dec6 P fd486120971d4be18bf0a84c60dd6394: Time spent starting tablet: real 0.014s user 0.004s sys 0.011s
I20260925 04:54:55.770671 10878 raft_consensus.cc:482] T 134a9e65efd248c7b344a3c051a9dec6 P e65d7ed29def474494a2f6fb87074c06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:54:55.771009 10878 raft_consensus.cc:504] T 134a9e65efd248c7b344a3c051a9dec6 P e65d7ed29def474494a2f6fb87074c06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd486120971d4be18bf0a84c60dd6394" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 39157 } } peers { permanent_uuid: "14e3e6ef72544f309b3fc3d9d59b6aa7" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 44315 } } peers { permanent_uuid: "e65d7ed29def474494a2f6fb87074c06" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 46111 } }
I20260925 04:54:55.772785 10878 leader_election.cc:288] T 134a9e65efd248c7b344a3c051a9dec6 P e65d7ed29def474494a2f6fb87074c06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd486120971d4be18bf0a84c60dd6394 (127.6.29.130:39157), 14e3e6ef72544f309b3fc3d9d59b6aa7 (127.6.29.129:44315)
I20260925 04:54:55.779488 10867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:55.773632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:35853 (local address 127.6.29.130:39157)
0925 04:54:55.773871 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:55.773880 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:55.773903 (+ 23us) server_negotiation.cc:407] Connection header received
0925 04:54:55.773980 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:55.773985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:55.774062 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:55.774164 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:54:55.774663 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:55.775637 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:55.777486 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:55.778525 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:55.778664 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:55.778721 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:55.779093 (+ 372us) server_negotiation.cc:299] Negotiation successful
0925 04:54:55.779198 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 04:54:55.779507 10855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:55.773149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:39157 (local address 127.6.29.131:35853)
0925 04:54:55.773378 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:55.773399 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:55.773539 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:55.774202 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:55.774206 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:55.774252 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:54:55.774492 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:55.774503 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:55.775791 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:55.775797 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:55.777324 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:55.777336 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:55.778691 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:55.778697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:55.778909 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:55.778951 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 04:54:55.779074 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 04:54:55.779223 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"spinlock_wait_cycles":33024}
I20260925 04:54:55.780153 10868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:55.773331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:41073 (local address 127.6.29.129:44315)
0925 04:54:55.773562 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:55.773569 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:55.774269 (+ 700us) server_negotiation.cc:407] Connection header received
0925 04:54:55.774460 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:55.774466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:55.774542 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:55.774632 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:54:55.775255 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:55.776198 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:55.778133 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:55.779159 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:55.779297 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:55.779366 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:55.779752 (+ 386us) server_negotiation.cc:299] Negotiation successful
0925 04:54:55.779921 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260925 04:54:55.780669 10880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:55.773416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:44315 (local address 127.6.29.131:41073)
0925 04:54:55.774169 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:55.774186 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:55.774301 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:55.774662 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:55.774667 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:55.774743 (+ 76us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:54:55.775106 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:55.775116 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:55.776354 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:55.776358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:55.777977 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:55.777990 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:55.779346 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:55.779355 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:55.779536 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:55.779574 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 04:54:55.780331 (+ 757us) client_negotiation.cc:240] Negotiation successful
0925 04:54:55.780444 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":420,"threads_started":1}
I20260925 04:54:55.780548 10675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "134a9e65efd248c7b344a3c051a9dec6" candidate_uuid: "e65d7ed29def474494a2f6fb87074c06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd486120971d4be18bf0a84c60dd6394" is_pre_election: true
I20260925 04:54:55.781101 10675 raft_consensus.cc:2454] T 134a9e65efd248c7b344a3c051a9dec6 P fd486120971d4be18bf0a84c60dd6394 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e65d7ed29def474494a2f6fb87074c06 in term 0.
I20260925 04:54:55.781666 10542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "134a9e65efd248c7b344a3c051a9dec6" candidate_uuid: "e65d7ed29def474494a2f6fb87074c06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14e3e6ef72544f309b3fc3d9d59b6aa7" is_pre_election: true
I20260925 04:54:55.781790 10741 leader_election.cc:302] T 134a9e65efd248c7b344a3c051a9dec6 P e65d7ed29def474494a2f6fb87074c06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e65d7ed29def474494a2f6fb87074c06, fd486120971d4be18bf0a84c60dd6394; no voters:
I20260925 04:54:55.782197 10878 raft_consensus.cc:2781] T 134a9e65efd248c7b344a3c051a9dec6 P e65d7ed29def474494a2f6fb87074c06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:54:55.782163 10542 raft_consensus.cc:2454] T 134a9e65efd248c7b344a3c051a9dec6 P 14e3e6ef72544f309b3fc3d9d59b6aa7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e65d7ed29def474494a2f6fb87074c06 in term 0.
I20260925 04:54:55.782344 10878 raft_consensus.cc:482] T 134a9e65efd248c7b344a3c051a9dec6 P e65d7ed29def474494a2f6fb87074c06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:54:55.782452 10878 raft_consensus.cc:3037] T 134a9e65efd248c7b344a3c051a9dec6 P e65d7ed29def474494a2f6fb87074c06 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:55.785130 10878 raft_consensus.cc:504] T 134a9e65efd248c7b344a3c051a9dec6 P e65d7ed29def474494a2f6fb87074c06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd486120971d4be18bf0a84c60dd6394" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 39157 } } peers { permanent_uuid: "14e3e6ef72544f309b3fc3d9d59b6aa7" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 44315 } } peers { permanent_uuid: "e65d7ed29def474494a2f6fb87074c06" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 46111 } }
I20260925 04:54:55.785863 10878 leader_election.cc:288] T 134a9e65efd248c7b344a3c051a9dec6 P e65d7ed29def474494a2f6fb87074c06 [CANDIDATE]: Term 1 election: Requested vote from peers fd486120971d4be18bf0a84c60dd6394 (127.6.29.130:39157), 14e3e6ef72544f309b3fc3d9d59b6aa7 (127.6.29.129:44315)
I20260925 04:54:55.786341 10542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "134a9e65efd248c7b344a3c051a9dec6" candidate_uuid: "e65d7ed29def474494a2f6fb87074c06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14e3e6ef72544f309b3fc3d9d59b6aa7"
I20260925 04:54:55.786375 10675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "134a9e65efd248c7b344a3c051a9dec6" candidate_uuid: "e65d7ed29def474494a2f6fb87074c06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd486120971d4be18bf0a84c60dd6394"
I20260925 04:54:55.786567 10542 raft_consensus.cc:3037] T 134a9e65efd248c7b344a3c051a9dec6 P 14e3e6ef72544f309b3fc3d9d59b6aa7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:55.786580 10675 raft_consensus.cc:3037] T 134a9e65efd248c7b344a3c051a9dec6 P fd486120971d4be18bf0a84c60dd6394 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:55.788833 10542 raft_consensus.cc:2454] T 134a9e65efd248c7b344a3c051a9dec6 P 14e3e6ef72544f309b3fc3d9d59b6aa7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e65d7ed29def474494a2f6fb87074c06 in term 1.
I20260925 04:54:55.788938 10675 raft_consensus.cc:2454] T 134a9e65efd248c7b344a3c051a9dec6 P fd486120971d4be18bf0a84c60dd6394 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e65d7ed29def474494a2f6fb87074c06 in term 1.
I20260925 04:54:55.789251 10743 leader_election.cc:302] T 134a9e65efd248c7b344a3c051a9dec6 P e65d7ed29def474494a2f6fb87074c06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14e3e6ef72544f309b3fc3d9d59b6aa7, e65d7ed29def474494a2f6fb87074c06; no voters:
I20260925 04:54:55.789631 10878 raft_consensus.cc:2781] T 134a9e65efd248c7b344a3c051a9dec6 P e65d7ed29def474494a2f6fb87074c06 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:54:55.790068 10878 raft_consensus.cc:686] T 134a9e65efd248c7b344a3c051a9dec6 P e65d7ed29def474494a2f6fb87074c06 [term 1 LEADER]: Becoming Leader. State: Replica: e65d7ed29def474494a2f6fb87074c06, State: Running, Role: LEADER
I20260925 04:54:55.790784 10878 consensus_queue.cc:237] T 134a9e65efd248c7b344a3c051a9dec6 P e65d7ed29def474494a2f6fb87074c06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd486120971d4be18bf0a84c60dd6394" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 39157 } } peers { permanent_uuid: "14e3e6ef72544f309b3fc3d9d59b6aa7" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 44315 } } peers { permanent_uuid: "e65d7ed29def474494a2f6fb87074c06" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 46111 } }
I20260925 04:54:55.795395 10399 catalog_manager.cc:5495] T 134a9e65efd248c7b344a3c051a9dec6 P e65d7ed29def474494a2f6fb87074c06 reported cstate change: term changed from 0 to 1, leader changed from <none> to e65d7ed29def474494a2f6fb87074c06 (127.6.29.131). New cstate: current_term: 1 leader_uuid: "e65d7ed29def474494a2f6fb87074c06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd486120971d4be18bf0a84c60dd6394" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 39157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14e3e6ef72544f309b3fc3d9d59b6aa7" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 44315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e65d7ed29def474494a2f6fb87074c06" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 46111 } health_report { overall_health: HEALTHY } } }
W20260925 04:54:55.801007 10721 tablet.cc:2369] T 134a9e65efd248c7b344a3c051a9dec6 P fd486120971d4be18bf0a84c60dd6394: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:54:55.806257 10588 tablet.cc:2369] T 134a9e65efd248c7b344a3c051a9dec6 P 14e3e6ef72544f309b3fc3d9d59b6aa7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:54:55.829241 10870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:55.823627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47942 (local address 127.6.29.131:46111)
0925 04:54:55.823920 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:55.823925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:55.823949 (+ 24us) server_negotiation.cc:407] Connection header received
0925 04:54:55.824182 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:55.824186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:55.824267 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:55.824401 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:54:55.824806 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:55.825534 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:55.826924 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:55.827325 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:55.827421 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:55.828077 (+ 656us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:54:55.828611 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:54:55.828735 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:55.828875 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 04:54:55.828944 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260925 04:54:55.835636 10853 tablet.cc:2369] T 134a9e65efd248c7b344a3c051a9dec6 P e65d7ed29def474494a2f6fb87074c06: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:54:55.837706 10542 raft_consensus.cc:1261] T 134a9e65efd248c7b344a3c051a9dec6 P 14e3e6ef72544f309b3fc3d9d59b6aa7 [term 1 FOLLOWER]: Refusing update from remote peer e65d7ed29def474494a2f6fb87074c06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:54:55.837729 10675 raft_consensus.cc:1261] T 134a9e65efd248c7b344a3c051a9dec6 P fd486120971d4be18bf0a84c60dd6394 [term 1 FOLLOWER]: Refusing update from remote peer e65d7ed29def474494a2f6fb87074c06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:54:55.838735 10881 consensus_queue.cc:1035] T 134a9e65efd248c7b344a3c051a9dec6 P e65d7ed29def474494a2f6fb87074c06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd486120971d4be18bf0a84c60dd6394" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 39157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:54:55.839082 10878 consensus_queue.cc:1035] T 134a9e65efd248c7b344a3c051a9dec6 P e65d7ed29def474494a2f6fb87074c06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14e3e6ef72544f309b3fc3d9d59b6aa7" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 44315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:54:55.851125 10884 mvcc.cc:205] Tried to move back new op lower bound from 7333118344540598272 to 7333118344362840064. Current Snapshot: MvccSnapshot[applied={T|T < 7333118344540598272}]
I20260925 04:54:55.852635 10885 mvcc.cc:205] Tried to move back new op lower bound from 7333118344540598272 to 7333118344362840064. Current Snapshot: MvccSnapshot[applied={T|T < 7333118344540598272}]
I20260925 04:54:55.865682 10398 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:54:55.866004 10398 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:54:55.869973 10398 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 79a1f9a2851e4c2d81586fc981aa5d53: Sending DeleteTablet for 3 replicas of tablet 134a9e65efd248c7b344a3c051a9dec6
I20260925 04:54:55.871299 10655 tablet_service.cc:1514] Processing DeleteTablet for tablet 134a9e65efd248c7b344a3c051a9dec6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:54:55 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:54172
I20260925 04:54:55.871299 10522 tablet_service.cc:1514] Processing DeleteTablet for tablet 134a9e65efd248c7b344a3c051a9dec6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:54:55 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:50194
I20260925 04:54:55.871341 10787 tablet_service.cc:1514] Processing DeleteTablet for tablet 134a9e65efd248c7b344a3c051a9dec6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:54:55 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:47934
I20260925 04:54:55.873771 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 10460
I20260925 04:54:55.888214 10894 tablet_replica.cc:330] T 134a9e65efd248c7b344a3c051a9dec6 P 14e3e6ef72544f309b3fc3d9d59b6aa7: stopping tablet replica
I20260925 04:54:55.888823 10894 raft_consensus.cc:2229] T 134a9e65efd248c7b344a3c051a9dec6 P 14e3e6ef72544f309b3fc3d9d59b6aa7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:54:55.889480 10894 raft_consensus.cc:2258] T 134a9e65efd248c7b344a3c051a9dec6 P 14e3e6ef72544f309b3fc3d9d59b6aa7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:54:55.890918 10893 tablet_replica.cc:330] T 134a9e65efd248c7b344a3c051a9dec6 P fd486120971d4be18bf0a84c60dd6394: stopping tablet replica
I20260925 04:54:55.891450 10893 raft_consensus.cc:2229] T 134a9e65efd248c7b344a3c051a9dec6 P fd486120971d4be18bf0a84c60dd6394 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:54:55.892107 10893 raft_consensus.cc:2258] T 134a9e65efd248c7b344a3c051a9dec6 P fd486120971d4be18bf0a84c60dd6394 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:54:55.893395 10895 tablet_replica.cc:330] T 134a9e65efd248c7b344a3c051a9dec6 P e65d7ed29def474494a2f6fb87074c06: stopping tablet replica
I20260925 04:54:55.893862 10895 raft_consensus.cc:2229] T 134a9e65efd248c7b344a3c051a9dec6 P e65d7ed29def474494a2f6fb87074c06 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:54:55.894599 10895 raft_consensus.cc:2258] T 134a9e65efd248c7b344a3c051a9dec6 P e65d7ed29def474494a2f6fb87074c06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:54:55.894807 10893 ts_tablet_manager.cc:1905] T 134a9e65efd248c7b344a3c051a9dec6 P fd486120971d4be18bf0a84c60dd6394: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:54:55.894851 10894 ts_tablet_manager.cc:1905] T 134a9e65efd248c7b344a3c051a9dec6 P 14e3e6ef72544f309b3fc3d9d59b6aa7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:54:55.896836 10895 ts_tablet_manager.cc:1905] T 134a9e65efd248c7b344a3c051a9dec6 P e65d7ed29def474494a2f6fb87074c06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:54:55.902117 10893 ts_tablet_manager.cc:1918] T 134a9e65efd248c7b344a3c051a9dec6 P fd486120971d4be18bf0a84c60dd6394: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:54:55.902315 10893 log.cc:1196] T 134a9e65efd248c7b344a3c051a9dec6 P fd486120971d4be18bf0a84c60dd6394: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-1/wal/wals/134a9e65efd248c7b344a3c051a9dec6
I20260925 04:54:55.902915 10893 ts_tablet_manager.cc:1939] T 134a9e65efd248c7b344a3c051a9dec6 P fd486120971d4be18bf0a84c60dd6394: Deleting consensus metadata
I20260925 04:54:55.904729 10895 ts_tablet_manager.cc:1918] T 134a9e65efd248c7b344a3c051a9dec6 P e65d7ed29def474494a2f6fb87074c06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:54:55.904912 10895 log.cc:1196] T 134a9e65efd248c7b344a3c051a9dec6 P e65d7ed29def474494a2f6fb87074c06: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-2/wal/wals/134a9e65efd248c7b344a3c051a9dec6
I20260925 04:54:55.905004 10384 catalog_manager.cc:4841] TS fd486120971d4be18bf0a84c60dd6394 (127.6.29.130:39157): tablet 134a9e65efd248c7b344a3c051a9dec6 (table test-table [id=7943d7f11a2a4690b75afd05da1f8d32]) successfully deleted
I20260925 04:54:55.905452 10895 ts_tablet_manager.cc:1939] T 134a9e65efd248c7b344a3c051a9dec6 P e65d7ed29def474494a2f6fb87074c06: Deleting consensus metadata
I20260925 04:54:55.907220 10384 catalog_manager.cc:4841] TS e65d7ed29def474494a2f6fb87074c06 (127.6.29.131:46111): tablet 134a9e65efd248c7b344a3c051a9dec6 (table test-table [id=7943d7f11a2a4690b75afd05da1f8d32]) successfully deleted
I20260925 04:54:55.908913 10894 ts_tablet_manager.cc:1918] T 134a9e65efd248c7b344a3c051a9dec6 P 14e3e6ef72544f309b3fc3d9d59b6aa7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:54:55.909122 10894 log.cc:1196] T 134a9e65efd248c7b344a3c051a9dec6 P 14e3e6ef72544f309b3fc3d9d59b6aa7: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790312053666503-6262-0/minicluster-data/ts-0/wal/wals/134a9e65efd248c7b344a3c051a9dec6
I20260925 04:54:55.909757 10894 ts_tablet_manager.cc:1939] T 134a9e65efd248c7b344a3c051a9dec6 P 14e3e6ef72544f309b3fc3d9d59b6aa7: Deleting consensus metadata
I20260925 04:54:55.911796 10386 catalog_manager.cc:4841] TS 14e3e6ef72544f309b3fc3d9d59b6aa7 (127.6.29.129:44315): tablet 134a9e65efd248c7b344a3c051a9dec6 (table test-table [id=7943d7f11a2a4690b75afd05da1f8d32]) successfully deleted
Sep 25 04:54:55 dist-test-slave-xzjr krb5kdc[10355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.129@KRBTEST.COM
I20260925 04:54:55.921229 10868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:55.875057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50198 (local address 127.6.29.129:44315)
0925 04:54:55.875252 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:55.875259 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:55.897670 (+ 22411us) server_negotiation.cc:407] Connection header received
0925 04:54:55.898358 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:55.898364 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:55.898453 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:55.898529 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:55.903271 (+ 4742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:55.904404 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:55.909136 (+ 4732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:55.909907 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:55.910016 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:55.914690 (+ 4674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:55.914703 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:55.914720 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:55.914756 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:55.917583 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:55.918299 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:55.918306 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:55.918309 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:55.918437 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:55.918829 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:55.918833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:55.918835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:55.919174 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:55.920670 (+ 1496us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:55.920803 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 04:54:55.920963 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 04:54:55.921996 10582 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:54:56.011837 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 10460
I20260925 04:54:56.035992 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 10593
Sep 25 04:54:56 dist-test-slave-xzjr krb5kdc[10355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.130@KRBTEST.COM
I20260925 04:54:56.053635 10867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:56.036924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54178 (local address 127.6.29.130:39157)
0925 04:54:56.037140 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:56.037147 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:56.037168 (+ 21us) server_negotiation.cc:407] Connection header received
0925 04:54:56.037290 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:56.037294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:56.037384 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:56.037491 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:56.038701 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:56.039438 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:56.040733 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:56.041150 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:56.041289 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:56.046059 (+ 4770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:56.046075 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:56.046091 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:56.046140 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:56.050235 (+ 4095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:56.051131 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:56.051141 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:56.051145 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:56.051296 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:56.051725 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:56.051730 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:56.051734 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:56.052225 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:56.052523 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:56.052868 (+ 345us) server_negotiation.cc:299] Negotiation successful
0925 04:54:56.053090 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 04:54:56.054280 10715 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:54:56.192901 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 10593
I20260925 04:54:56.209900 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 10725
Sep 25 04:54:56 dist-test-slave-xzjr krb5kdc[10355](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.131@KRBTEST.COM
I20260925 04:54:56.233665 10870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:56.210707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47956 (local address 127.6.29.131:46111)
0925 04:54:56.210881 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:56.210885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:56.210971 (+ 86us) server_negotiation.cc:407] Connection header received
0925 04:54:56.211178 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:56.211181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:56.211246 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:56.211328 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:56.213250 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:56.214122 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:56.216429 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:56.217448 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:56.217668 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:56.224072 (+ 6404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:56.224088 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:56.224109 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:54:56.224208 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:56.229571 (+ 5363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:56.230460 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:56.230469 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:56.230472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:56.230695 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:54:56.231174 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:54:56.231179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:54:56.231183 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:54:56.231645 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:56.231971 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:56.233206 (+ 1235us) server_negotiation.cc:299] Negotiation successful
0925 04:54:56.233375 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 04:54:56.234537 10847 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:54:56.358652 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 10725
I20260925 04:54:56.378480 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 10368
I20260925 04:54:56.379693 10429 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:54:56.487843 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 10368
2026-09-25T04:54:56Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4323 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T04:54:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:54:56Z Disabled control of system clock
I20260925 04:54:56.556257 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:39541
--webserver_interface=127.6.29.190
--webserver_port=0
--builtin_ntp_servers=127.6.29.148:45553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:39541
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 04:54:56.887516 10904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:54:56.887916 10904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:54:56.888067 10904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:54:56.895521 10904 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:54:56.895637 10904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:54:56.895713 10904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:54:56.895843 10904 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:54:56.895895 10904 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:54:56.905609 10904 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:45553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:39541
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:39541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:54:56.907025 10904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:54:56.908910 10904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:54:56.921022 10909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:56.921133 10912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:56.921154 10910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:54:56.922537 10904 server_base.cc:1034] running on GCE node
I20260925 04:54:56.923424 10904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:54:56.925015 10904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:54:56.926249 10904 hybrid_clock.cc:648] HybridClock initialized: now 1790312096926184 us; error 51 us; skew 500 ppm
I20260925 04:54:56.926644 10904 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 04:54:56.933117 10904 webserver.cc:460] Webserver started at http://127.6.29.190:46125/ using document root <none> and password file <none>
I20260925 04:54:56.933917 10904 fs_manager.cc:359] Metadata directory not provided
I20260925 04:54:56.934012 10904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:54:56.934351 10904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:54:56.937446 10904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "19489055a8934759b9fc934c52199e2c"
format_stamp: "Formatted at 2026-09-25 04:54:56 on dist-test-slave-xzjr"
server_key: "035caeb2769be7b54679814a6f5f827b"
server_key_iv: "6b2a8fd10bdd91b5a16706143c49c808"
server_key_version: "encryptionkey@0"
I20260925 04:54:56.938289 10904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "19489055a8934759b9fc934c52199e2c"
format_stamp: "Formatted at 2026-09-25 04:54:56 on dist-test-slave-xzjr"
server_key: "035caeb2769be7b54679814a6f5f827b"
server_key_iv: "6b2a8fd10bdd91b5a16706143c49c808"
server_key_version: "encryptionkey@0"
I20260925 04:54:56.943903 10904 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260925 04:54:56.947563 10918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:56.949208 10904 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:54:56.949450 10904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "19489055a8934759b9fc934c52199e2c"
format_stamp: "Formatted at 2026-09-25 04:54:56 on dist-test-slave-xzjr"
server_key: "035caeb2769be7b54679814a6f5f827b"
server_key_iv: "6b2a8fd10bdd91b5a16706143c49c808"
server_key_version: "encryptionkey@0"
I20260925 04:54:56.949697 10904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:54:56.956048 10904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:54:56.956883 10904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:54:56.957232 10904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:54:56.975732 10904 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:39541
I20260925 04:54:56.975814 10969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:39541 every 8 connection(s)
I20260925 04:54:56.977150 10904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:54:56.981012 10970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:54:56.985595 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 10904
I20260925 04:54:56.985841 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 04:54:56.986510 6262 external_mini_cluster.cc:1428] Setting key 297684985cb1cd9f6c53ab604575a851
I20260925 04:54:56.992365 10970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19489055a8934759b9fc934c52199e2c: Bootstrap starting.
I20260925 04:54:56.995160 10973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:56.988684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37182 (local address 127.6.29.190:39541)
0925 04:54:56.989791 (+ 1107us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:56.989802 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:56.989851 (+ 49us) server_negotiation.cc:407] Connection header received
0925 04:54:56.991493 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:56.991524 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:56.992041 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:56.992442 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:56.993512 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:56.993519 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:56.993532 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:54:56.993548 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:56.993603 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:54:56.993618 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:56.993864 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:56.994021 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 04:54:56.994261 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":406,"threads_started":1}
I20260925 04:54:56.996075 10970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19489055a8934759b9fc934c52199e2c: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:56.997233 10970 log.cc:824] T 00000000000000000000000000000000 P 19489055a8934759b9fc934c52199e2c: Log is configured to *not* fsync() on all Append() calls
I20260925 04:54:57.000370 10970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19489055a8934759b9fc934c52199e2c: No bootstrap required, opened a new log
I20260925 04:54:57.008155 10970 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19489055a8934759b9fc934c52199e2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19489055a8934759b9fc934c52199e2c" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39541 } }
I20260925 04:54:57.008528 10970 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19489055a8934759b9fc934c52199e2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:57.008646 10970 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19489055a8934759b9fc934c52199e2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19489055a8934759b9fc934c52199e2c, State: Initialized, Role: FOLLOWER
I20260925 04:54:57.009290 10970 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19489055a8934759b9fc934c52199e2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19489055a8934759b9fc934c52199e2c" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39541 } }
I20260925 04:54:57.009519 10970 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19489055a8934759b9fc934c52199e2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:54:57.009654 10970 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19489055a8934759b9fc934c52199e2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:54:57.009829 10970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19489055a8934759b9fc934c52199e2c [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:57.012036 10970 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19489055a8934759b9fc934c52199e2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19489055a8934759b9fc934c52199e2c" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39541 } }
I20260925 04:54:57.012557 10970 leader_election.cc:302] T 00000000000000000000000000000000 P 19489055a8934759b9fc934c52199e2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19489055a8934759b9fc934c52199e2c; no voters:
I20260925 04:54:57.013258 10970 leader_election.cc:288] T 00000000000000000000000000000000 P 19489055a8934759b9fc934c52199e2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:54:57.013391 10975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19489055a8934759b9fc934c52199e2c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:54:57.014083 10975 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19489055a8934759b9fc934c52199e2c [term 1 LEADER]: Becoming Leader. State: Replica: 19489055a8934759b9fc934c52199e2c, State: Running, Role: LEADER
I20260925 04:54:57.014681 10975 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19489055a8934759b9fc934c52199e2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19489055a8934759b9fc934c52199e2c" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39541 } }
I20260925 04:54:57.015609 10970 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19489055a8934759b9fc934c52199e2c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:54:57.017603 10976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19489055a8934759b9fc934c52199e2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19489055a8934759b9fc934c52199e2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19489055a8934759b9fc934c52199e2c" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39541 } } }
I20260925 04:54:57.017988 10976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19489055a8934759b9fc934c52199e2c [sys.catalog]: This master's current role is: LEADER
I20260925 04:54:57.017808 10977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19489055a8934759b9fc934c52199e2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19489055a8934759b9fc934c52199e2c. Latest consensus state: current_term: 1 leader_uuid: "19489055a8934759b9fc934c52199e2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19489055a8934759b9fc934c52199e2c" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39541 } } }
I20260925 04:54:57.018218 10977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19489055a8934759b9fc934c52199e2c [sys.catalog]: This master's current role is: LEADER
I20260925 04:54:57.020321 10982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:54:57.026110 10982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:54:57.036204 10982 catalog_manager.cc:1347] Generated new cluster ID: a819a5eddfdc44fd8726ed01ba2519f3
I20260925 04:54:57.036348 10982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:54:57.048630 10982 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:54:57.049072 10982 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:54:57.057299 10982 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19489055a8934759b9fc934c52199e2c: Generated new TSK 0
I20260925 04:54:57.058214 10982 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:54:57.068816 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.129:0
--local_ip_for_outbound_sockets=127.6.29.129
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:39541
--builtin_ntp_servers=127.6.29.148:45553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 04:54:57.426105 10994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:54:57.426426 10994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:54:57.426532 10994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:54:57.433928 10994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:54:57.434036 10994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:54:57.434149 10994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.129
I20260925 04:54:57.443977 10994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:45553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:39541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.129
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:54:57.445267 10994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:54:57.447017 10994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:54:57.462577 10999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:57.462659 11000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:57.462613 11002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:54:57.465381 10994 server_base.cc:1034] running on GCE node
I20260925 04:54:57.466054 10994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:54:57.467198 10994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:54:57.468403 10994 hybrid_clock.cc:648] HybridClock initialized: now 1790312097468348 us; error 46 us; skew 500 ppm
I20260925 04:54:57.468752 10994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 04:54:57.471225 10994 webserver.cc:460] Webserver started at http://127.6.29.129:45583/ using document root <none> and password file <none>
I20260925 04:54:57.472054 10994 fs_manager.cc:359] Metadata directory not provided
I20260925 04:54:57.472146 10994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:54:57.472535 10994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:54:57.475467 10994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-0/data/instance:
uuid: "b80ee883e0fe4270a3a641a60c8dcc72"
format_stamp: "Formatted at 2026-09-25 04:54:57 on dist-test-slave-xzjr"
server_key: "2a1398e19a929fc35e1a5b47ebf09f77"
server_key_iv: "af03f0acd57b687e2c8ab6f4bc433e0b"
server_key_version: "encryptionkey@0"
I20260925 04:54:57.476329 10994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance:
uuid: "b80ee883e0fe4270a3a641a60c8dcc72"
format_stamp: "Formatted at 2026-09-25 04:54:57 on dist-test-slave-xzjr"
server_key: "2a1398e19a929fc35e1a5b47ebf09f77"
server_key_iv: "af03f0acd57b687e2c8ab6f4bc433e0b"
server_key_version: "encryptionkey@0"
I20260925 04:54:57.481648 10994 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 04:54:57.485543 11008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:57.487068 10994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 04:54:57.487282 10994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-0/wal
uuid: "b80ee883e0fe4270a3a641a60c8dcc72"
format_stamp: "Formatted at 2026-09-25 04:54:57 on dist-test-slave-xzjr"
server_key: "2a1398e19a929fc35e1a5b47ebf09f77"
server_key_iv: "af03f0acd57b687e2c8ab6f4bc433e0b"
server_key_version: "encryptionkey@0"
I20260925 04:54:57.487543 10994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:54:57.493590 10994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:54:57.494539 10994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:54:57.494839 10994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:54:57.495895 10994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:54:57.497380 10994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:54:57.497505 10994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:57.497638 10994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:54:57.497736 10994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:57.533424 10994 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.129:36591
I20260925 04:54:57.533525 11120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.129:36591 every 8 connection(s)
I20260925 04:54:57.534801 10994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
I20260925 04:54:57.543358 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:57.537446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:57405 (local address 127.6.29.190:39541)
0925 04:54:57.537892 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:57.537896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:57.539195 (+ 1299us) server_negotiation.cc:407] Connection header received
0925 04:54:57.541007 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:57.541012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:57.541082 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:57.541214 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:57.542594 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:57.542600 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:57.542611 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:54:57.542628 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:57.542675 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:54:57.542687 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:57.542810 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:57.543004 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 04:54:57.543070 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":242,"threads_started":1}
I20260925 04:54:57.544143 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 10994
I20260925 04:54:57.544358 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance
I20260925 04:54:57.544682 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:57.538084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39541 (local address 127.6.29.129:57405)
0925 04:54:57.539017 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:57.539064 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:57.540724 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:57.541450 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:57.541459 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:57.541555 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:54:57.542284 (+ 729us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:54:57.542286 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:54:57.542324 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:54:57.542413 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:54:57.542793 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:54:57.542802 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:54:57.542810 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 04:54:57.543092 (+ 282us) client_negotiation.cc:240] Negotiation successful
0925 04:54:57.543359 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":270,"threads_started":1}
I20260925 04:54:57.545145 6262 external_mini_cluster.cc:1428] Setting key 0039b2cbb0b8b5e97430716dc1dab55d
I20260925 04:54:57.546466 11121 heartbeater.cc:347] Connected to a master server at 127.6.29.190:39541
I20260925 04:54:57.546788 11121 heartbeater.cc:464] Registering TS with master...
I20260925 04:54:57.547428 11121 heartbeater.cc:511] Master 127.6.29.190:39541 requested a full tablet report, sending...
I20260925 04:54:57.548364 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.130:0
--local_ip_for_outbound_sockets=127.6.29.130
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:39541
--builtin_ntp_servers=127.6.29.148:45553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 04:54:57.549894 10935 ts_manager.cc:194] Registered new tserver with Master: b80ee883e0fe4270a3a641a60c8dcc72 (127.6.29.129:36591)
W20260925 04:54:57.883016 11125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:54:57.883368 11125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:54:57.883529 11125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:54:57.891113 11125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:54:57.891227 11125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:54:57.891338 11125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.130
I20260925 04:54:57.900964 11125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:45553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:39541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.130
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:54:57.902273 11125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:54:57.904134 11125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:54:57.921716 11133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:57.922767 11130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:54:57.923097 11125 server_base.cc:1034] running on GCE node
W20260925 04:54:57.922255 11131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:54:57.924984 11125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:54:57.926375 11125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:54:57.927641 11125 hybrid_clock.cc:648] HybridClock initialized: now 1790312097927465 us; error 151 us; skew 500 ppm
I20260925 04:54:57.928015 11125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 04:54:57.931293 11125 webserver.cc:460] Webserver started at http://127.6.29.130:44981/ using document root <none> and password file <none>
I20260925 04:54:57.932185 11125 fs_manager.cc:359] Metadata directory not provided
I20260925 04:54:57.932318 11125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:54:57.932641 11125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:54:57.936944 11125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-1/data/instance:
uuid: "a28b313a46aa459d822df7ecd20cb321"
format_stamp: "Formatted at 2026-09-25 04:54:57 on dist-test-slave-xzjr"
server_key: "6991018ce22241496710a8e59df7c2f2"
server_key_iv: "a50eb3d27aa5cb0f27b5a99a8ca3aff1"
server_key_version: "encryptionkey@0"
I20260925 04:54:57.938136 11125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance:
uuid: "a28b313a46aa459d822df7ecd20cb321"
format_stamp: "Formatted at 2026-09-25 04:54:57 on dist-test-slave-xzjr"
server_key: "6991018ce22241496710a8e59df7c2f2"
server_key_iv: "a50eb3d27aa5cb0f27b5a99a8ca3aff1"
server_key_version: "encryptionkey@0"
I20260925 04:54:57.945405 11125 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260925 04:54:57.949587 11139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:57.951593 11125 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:54:57.951824 11125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-1/wal
uuid: "a28b313a46aa459d822df7ecd20cb321"
format_stamp: "Formatted at 2026-09-25 04:54:57 on dist-test-slave-xzjr"
server_key: "6991018ce22241496710a8e59df7c2f2"
server_key_iv: "a50eb3d27aa5cb0f27b5a99a8ca3aff1"
server_key_version: "encryptionkey@0"
I20260925 04:54:57.952097 11125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:54:57.960155 11125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:54:57.961097 11125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:54:57.961402 11125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:54:57.962371 11125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:54:57.963934 11125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:54:57.964022 11125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:57.964159 11125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:54:57.964242 11125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:58.000726 11125 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.130:46029
I20260925 04:54:58.000806 11251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.130:46029 every 8 connection(s)
I20260925 04:54:58.002111 11125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
I20260925 04:54:58.010003 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 11125
I20260925 04:54:58.010252 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance
I20260925 04:54:58.010623 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:58.004903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:45237 (local address 127.6.29.190:39541)
0925 04:54:58.005076 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:58.005080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:58.006536 (+ 1456us) server_negotiation.cc:407] Connection header received
0925 04:54:58.008311 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:58.008317 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:58.008403 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:58.008520 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:58.009843 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:58.009849 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:58.009854 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:54:58.009872 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:58.009909 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:54:58.009920 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:58.010050 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:58.010271 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 04:54:58.010342 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 04:54:58.010805 6262 external_mini_cluster.cc:1428] Setting key 43bb2ba6c8086b634d3a82cfb7dde8d8
I20260925 04:54:58.011534 11254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:58.005392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39541 (local address 127.6.29.130:45237)
0925 04:54:58.006368 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:58.006404 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:58.008033 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:58.008753 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:58.008762 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:58.008855 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:54:58.009546 (+ 691us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:54:58.009549 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:54:58.009585 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:54:58.009670 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:54:58.010011 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:54:58.010021 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:54:58.010030 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 04:54:58.010298 (+ 268us) client_negotiation.cc:240] Negotiation successful
0925 04:54:58.010546 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":319,"threads_started":1}
I20260925 04:54:58.012985 11252 heartbeater.cc:347] Connected to a master server at 127.6.29.190:39541
I20260925 04:54:58.013283 11252 heartbeater.cc:464] Registering TS with master...
I20260925 04:54:58.013669 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.131:0
--local_ip_for_outbound_sockets=127.6.29.131
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:39541
--builtin_ntp_servers=127.6.29.148:45553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 04:54:58.013947 11252 heartbeater.cc:511] Master 127.6.29.190:39541 requested a full tablet report, sending...
I20260925 04:54:58.015357 10934 ts_manager.cc:194] Registered new tserver with Master: a28b313a46aa459d822df7ecd20cb321 (127.6.29.130:46029)
W20260925 04:54:58.349018 11255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:54:58.349306 11255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:54:58.349470 11255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:54:58.357074 11255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:54:58.357187 11255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:54:58.357412 11255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.131
I20260925 04:54:58.366755 11255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:45553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:39541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.131
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:54:58.368117 11255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:54:58.370018 11255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:54:58.383936 11260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:58.383960 11261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:58.383991 11263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:54:58.384692 11255 server_base.cc:1034] running on GCE node
I20260925 04:54:58.385329 11255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:54:58.386371 11255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:54:58.387600 11255 hybrid_clock.cc:648] HybridClock initialized: now 1790312098387553 us; error 41 us; skew 500 ppm
I20260925 04:54:58.387980 11255 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 04:54:58.390378 11255 webserver.cc:460] Webserver started at http://127.6.29.131:33277/ using document root <none> and password file <none>
I20260925 04:54:58.391134 11255 fs_manager.cc:359] Metadata directory not provided
I20260925 04:54:58.391271 11255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:54:58.391614 11255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:54:58.394728 11255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-2/data/instance:
uuid: "438e234cf3d64a5890b7daad13a4fc40"
format_stamp: "Formatted at 2026-09-25 04:54:58 on dist-test-slave-xzjr"
server_key: "36b7c6997822a6f060aa3060f1a3f899"
server_key_iv: "50fde2f3e38ade27180fbc8e81f34303"
server_key_version: "encryptionkey@0"
I20260925 04:54:58.395543 11255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance:
uuid: "438e234cf3d64a5890b7daad13a4fc40"
format_stamp: "Formatted at 2026-09-25 04:54:58 on dist-test-slave-xzjr"
server_key: "36b7c6997822a6f060aa3060f1a3f899"
server_key_iv: "50fde2f3e38ade27180fbc8e81f34303"
server_key_version: "encryptionkey@0"
I20260925 04:54:58.400976 11255 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 04:54:58.404754 11269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:58.406247 11255 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:54:58.406438 11255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-2/wal
uuid: "438e234cf3d64a5890b7daad13a4fc40"
format_stamp: "Formatted at 2026-09-25 04:54:58 on dist-test-slave-xzjr"
server_key: "36b7c6997822a6f060aa3060f1a3f899"
server_key_iv: "50fde2f3e38ade27180fbc8e81f34303"
server_key_version: "encryptionkey@0"
I20260925 04:54:58.406687 11255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:54:58.412662 11255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:54:58.413551 11255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:54:58.413828 11255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:54:58.414804 11255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:54:58.416287 11255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:54:58.416402 11255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:58.416536 11255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:54:58.416626 11255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:58.449608 11255 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.131:34189
I20260925 04:54:58.449659 11381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.131:34189 every 8 connection(s)
I20260925 04:54:58.450942 11255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
I20260925 04:54:58.454057 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 11255
I20260925 04:54:58.454247 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance
I20260925 04:54:58.454867 6262 external_mini_cluster.cc:1428] Setting key 1c9decb352088cda4a801a4adb89d2b3
I20260925 04:54:58.460415 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:58.453532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:35855 (local address 127.6.29.190:39541)
0925 04:54:58.453728 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:58.453733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:58.455567 (+ 1834us) server_negotiation.cc:407] Connection header received
0925 04:54:58.458013 (+ 2446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:58.458019 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:58.458121 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:58.458247 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:58.459551 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:58.459557 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:58.459562 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:54:58.459580 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:58.459619 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:54:58.459632 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:58.459862 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:58.460085 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 04:54:58.460150 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 04:54:58.461373 11384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:58.454280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39541 (local address 127.6.29.131:35855)
0925 04:54:58.455353 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:58.455431 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:58.457682 (+ 2251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:58.458487 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:58.458496 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:58.458610 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:54:58.459243 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:54:58.459245 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:54:58.459283 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:54:58.459370 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:54:58.459843 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:54:58.459854 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:54:58.459862 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 04:54:58.460173 (+ 311us) client_negotiation.cc:240] Negotiation successful
0925 04:54:58.460442 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":332,"threads_started":1}
I20260925 04:54:58.462747 11382 heartbeater.cc:347] Connected to a master server at 127.6.29.190:39541
I20260925 04:54:58.463064 11382 heartbeater.cc:464] Registering TS with master...
I20260925 04:54:58.463662 11382 heartbeater.cc:511] Master 127.6.29.190:39541 requested a full tablet report, sending...
I20260925 04:54:58.465013 10934 ts_manager.cc:194] Registered new tserver with Master: 438e234cf3d64a5890b7daad13a4fc40 (127.6.29.131:34189)
I20260925 04:54:58.469991 6262 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:54:58.476696 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:58.474329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37192 (local address 127.6.29.190:39541)
0925 04:54:58.474505 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:58.474510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:58.475067 (+ 557us) server_negotiation.cc:407] Connection header received
0925 04:54:58.475191 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:58.475195 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:58.475246 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:58.475333 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:58.476249 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:58.476253 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:58.476256 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:54:58.476269 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:58.476295 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:54:58.476305 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:58.476404 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:58.476445 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 04:54:58.476486 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 04:54:58.480939 10934 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37192:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:54:58.487954 10934 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:54:58.510264 11394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:58.500017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.131:34189 (local address 127.0.0.1:45264)
0925 04:54:58.503076 (+ 3059us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:58.503131 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:58.503280 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:58.507012 (+ 3732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:58.507023 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:58.507141 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:54:58.508083 (+ 942us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:54:58.508087 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:54:58.508135 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:54:58.508224 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:54:58.509472 (+ 1248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:54:58.509482 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:54:58.509490 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 04:54:58.509675 (+ 185us) client_negotiation.cc:240] Negotiation successful
0925 04:54:58.509778 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2776,"thread_start_us":3399,"threads_started":1}
I20260925 04:54:58.510442 11397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:58.500743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43034 (local address 127.6.29.129:36591)
0925 04:54:58.502375 (+ 1632us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:58.502386 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:58.506578 (+ 4192us) server_negotiation.cc:407] Connection header received
0925 04:54:58.506891 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:58.506903 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:58.507139 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:58.507282 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:58.509424 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:58.509435 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:58.509448 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:54:58.509477 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:58.509534 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:54:58.509555 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:58.509690 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:58.510002 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 04:54:58.510112 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1453,"thread_start_us":1299,"threads_started":1}
I20260925 04:54:58.510759 11396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:58.500150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45264 (local address 127.6.29.131:34189)
0925 04:54:58.505834 (+ 5684us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:58.505847 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:58.505887 (+ 40us) server_negotiation.cc:407] Connection header received
0925 04:54:58.505974 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:58.505986 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:58.506279 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:58.507692 (+ 1413us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:58.509227 (+ 1535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:58.509237 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:58.509254 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:54:58.509275 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:58.509335 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:54:58.509360 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:58.509514 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:58.509820 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 04:54:58.509935 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5481,"thread_start_us":5305,"threads_started":1}
I20260925 04:54:58.511020 11395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:58.502248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:46029 (local address 127.0.0.1:46526)
0925 04:54:58.507140 (+ 4892us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:58.507157 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:58.507269 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:58.508802 (+ 1533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:58.508806 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:58.508828 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:54:58.509821 (+ 993us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:54:58.509824 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:54:58.509838 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:54:58.509874 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:54:58.510394 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:54:58.510399 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:54:58.510404 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 04:54:58.510545 (+ 141us) client_negotiation.cc:240] Negotiation successful
0925 04:54:58.510591 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4789,"thread_start_us":3780,"threads_started":1}
I20260925 04:54:58.511286 11398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:58.500085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:36591 (local address 127.0.0.1:43034)
0925 04:54:58.506417 (+ 6332us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:58.506443 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:58.506699 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:58.508362 (+ 1663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:58.508366 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:58.508389 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:54:58.509217 (+ 828us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:54:58.509219 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:54:58.509230 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:54:58.509263 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:54:58.509709 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:54:58.509712 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:54:58.509717 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 04:54:58.510737 (+ 1020us) client_negotiation.cc:240] Negotiation successful
0925 04:54:58.510778 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6150,"thread_start_us":6811,"threads_started":1}
I20260925 04:54:58.512212 11399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:58.502827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46526 (local address 127.6.29.130:46029)
0925 04:54:58.506845 (+ 4018us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:58.506856 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:58.507270 (+ 414us) server_negotiation.cc:407] Connection header received
0925 04:54:58.507398 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:58.507406 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:58.507594 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:58.508124 (+ 530us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:58.510043 (+ 1919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:58.510053 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:58.510068 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:54:58.510089 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:58.510143 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:54:58.510178 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:58.510341 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:58.511251 (+ 910us) server_negotiation.cc:299] Negotiation successful
0925 04:54:58.511890 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3855,"thread_start_us":3747,"threads_started":1}
I20260925 04:54:58.514906 11187 tablet_service.cc:1467] Processing CreateTablet for tablet 9b1615195a4147019818c46ba67a9e4e (DEFAULT_TABLE table=test-table [id=b884da746c294da7a89965e914d08bbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:54:58.514916 11317 tablet_service.cc:1467] Processing CreateTablet for tablet 9b1615195a4147019818c46ba67a9e4e (DEFAULT_TABLE table=test-table [id=b884da746c294da7a89965e914d08bbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:54:58.514916 11056 tablet_service.cc:1467] Processing CreateTablet for tablet 9b1615195a4147019818c46ba67a9e4e (DEFAULT_TABLE table=test-table [id=b884da746c294da7a89965e914d08bbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:54:58.516283 11187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b1615195a4147019818c46ba67a9e4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:54:58.516369 11056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b1615195a4147019818c46ba67a9e4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:54:58.516362 11317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b1615195a4147019818c46ba67a9e4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:54:58.527958 11400 tablet_bootstrap.cc:492] T 9b1615195a4147019818c46ba67a9e4e P a28b313a46aa459d822df7ecd20cb321: Bootstrap starting.
I20260925 04:54:58.530951 11402 tablet_bootstrap.cc:492] T 9b1615195a4147019818c46ba67a9e4e P 438e234cf3d64a5890b7daad13a4fc40: Bootstrap starting.
I20260925 04:54:58.531606 11400 tablet_bootstrap.cc:654] T 9b1615195a4147019818c46ba67a9e4e P a28b313a46aa459d822df7ecd20cb321: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:58.532785 11401 tablet_bootstrap.cc:492] T 9b1615195a4147019818c46ba67a9e4e P b80ee883e0fe4270a3a641a60c8dcc72: Bootstrap starting.
I20260925 04:54:58.534541 11400 log.cc:824] T 9b1615195a4147019818c46ba67a9e4e P a28b313a46aa459d822df7ecd20cb321: Log is configured to *not* fsync() on all Append() calls
I20260925 04:54:58.534621 11402 tablet_bootstrap.cc:654] T 9b1615195a4147019818c46ba67a9e4e P 438e234cf3d64a5890b7daad13a4fc40: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:58.536041 11402 log.cc:824] T 9b1615195a4147019818c46ba67a9e4e P 438e234cf3d64a5890b7daad13a4fc40: Log is configured to *not* fsync() on all Append() calls
I20260925 04:54:58.536288 11401 tablet_bootstrap.cc:654] T 9b1615195a4147019818c46ba67a9e4e P b80ee883e0fe4270a3a641a60c8dcc72: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:58.537449 11401 log.cc:824] T 9b1615195a4147019818c46ba67a9e4e P b80ee883e0fe4270a3a641a60c8dcc72: Log is configured to *not* fsync() on all Append() calls
I20260925 04:54:58.545691 11402 tablet_bootstrap.cc:492] T 9b1615195a4147019818c46ba67a9e4e P 438e234cf3d64a5890b7daad13a4fc40: No bootstrap required, opened a new log
I20260925 04:54:58.545691 11400 tablet_bootstrap.cc:492] T 9b1615195a4147019818c46ba67a9e4e P a28b313a46aa459d822df7ecd20cb321: No bootstrap required, opened a new log
I20260925 04:54:58.546015 11402 ts_tablet_manager.cc:1397] T 9b1615195a4147019818c46ba67a9e4e P 438e234cf3d64a5890b7daad13a4fc40: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260925 04:54:58.546015 11400 ts_tablet_manager.cc:1397] T 9b1615195a4147019818c46ba67a9e4e P a28b313a46aa459d822df7ecd20cb321: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.007s
I20260925 04:54:58.553032 11121 heartbeater.cc:503] Master 127.6.29.190:39541 was elected leader, sending a full tablet report...
I20260925 04:54:58.556960 11402 raft_consensus.cc:348] T 9b1615195a4147019818c46ba67a9e4e P 438e234cf3d64a5890b7daad13a4fc40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "438e234cf3d64a5890b7daad13a4fc40" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 34189 } } peers { permanent_uuid: "b80ee883e0fe4270a3a641a60c8dcc72" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 36591 } } peers { permanent_uuid: "a28b313a46aa459d822df7ecd20cb321" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 46029 } }
I20260925 04:54:58.557477 11402 raft_consensus.cc:374] T 9b1615195a4147019818c46ba67a9e4e P 438e234cf3d64a5890b7daad13a4fc40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:58.557590 11402 raft_consensus.cc:729] T 9b1615195a4147019818c46ba67a9e4e P 438e234cf3d64a5890b7daad13a4fc40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 438e234cf3d64a5890b7daad13a4fc40, State: Initialized, Role: FOLLOWER
I20260925 04:54:58.558495 11402 consensus_queue.cc:260] T 9b1615195a4147019818c46ba67a9e4e P 438e234cf3d64a5890b7daad13a4fc40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "438e234cf3d64a5890b7daad13a4fc40" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 34189 } } peers { permanent_uuid: "b80ee883e0fe4270a3a641a60c8dcc72" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 36591 } } peers { permanent_uuid: "a28b313a46aa459d822df7ecd20cb321" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 46029 } }
I20260925 04:54:58.558622 11400 raft_consensus.cc:348] T 9b1615195a4147019818c46ba67a9e4e P a28b313a46aa459d822df7ecd20cb321 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "438e234cf3d64a5890b7daad13a4fc40" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 34189 } } peers { permanent_uuid: "b80ee883e0fe4270a3a641a60c8dcc72" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 36591 } } peers { permanent_uuid: "a28b313a46aa459d822df7ecd20cb321" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 46029 } }
I20260925 04:54:58.559341 11400 raft_consensus.cc:374] T 9b1615195a4147019818c46ba67a9e4e P a28b313a46aa459d822df7ecd20cb321 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:58.559510 11400 raft_consensus.cc:729] T 9b1615195a4147019818c46ba67a9e4e P a28b313a46aa459d822df7ecd20cb321 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a28b313a46aa459d822df7ecd20cb321, State: Initialized, Role: FOLLOWER
I20260925 04:54:58.560493 11400 consensus_queue.cc:260] T 9b1615195a4147019818c46ba67a9e4e P a28b313a46aa459d822df7ecd20cb321 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "438e234cf3d64a5890b7daad13a4fc40" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 34189 } } peers { permanent_uuid: "b80ee883e0fe4270a3a641a60c8dcc72" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 36591 } } peers { permanent_uuid: "a28b313a46aa459d822df7ecd20cb321" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 46029 } }
I20260925 04:54:58.561758 11252 heartbeater.cc:503] Master 127.6.29.190:39541 was elected leader, sending a full tablet report...
I20260925 04:54:58.561865 11401 tablet_bootstrap.cc:492] T 9b1615195a4147019818c46ba67a9e4e P b80ee883e0fe4270a3a641a60c8dcc72: No bootstrap required, opened a new log
I20260925 04:54:58.562162 11401 ts_tablet_manager.cc:1397] T 9b1615195a4147019818c46ba67a9e4e P b80ee883e0fe4270a3a641a60c8dcc72: Time spent bootstrapping tablet: real 0.030s user 0.018s sys 0.009s
I20260925 04:54:58.562718 11400 ts_tablet_manager.cc:1428] T 9b1615195a4147019818c46ba67a9e4e P a28b313a46aa459d822df7ecd20cb321: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
I20260925 04:54:58.563637 11382 heartbeater.cc:503] Master 127.6.29.190:39541 was elected leader, sending a full tablet report...
I20260925 04:54:58.564431 11402 ts_tablet_manager.cc:1428] T 9b1615195a4147019818c46ba67a9e4e P 438e234cf3d64a5890b7daad13a4fc40: Time spent starting tablet: real 0.018s user 0.016s sys 0.004s
I20260925 04:54:58.570778 11401 raft_consensus.cc:348] T 9b1615195a4147019818c46ba67a9e4e P b80ee883e0fe4270a3a641a60c8dcc72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "438e234cf3d64a5890b7daad13a4fc40" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 34189 } } peers { permanent_uuid: "b80ee883e0fe4270a3a641a60c8dcc72" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 36591 } } peers { permanent_uuid: "a28b313a46aa459d822df7ecd20cb321" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 46029 } }
I20260925 04:54:58.571226 11401 raft_consensus.cc:374] T 9b1615195a4147019818c46ba67a9e4e P b80ee883e0fe4270a3a641a60c8dcc72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:58.571328 11401 raft_consensus.cc:729] T 9b1615195a4147019818c46ba67a9e4e P b80ee883e0fe4270a3a641a60c8dcc72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b80ee883e0fe4270a3a641a60c8dcc72, State: Initialized, Role: FOLLOWER
I20260925 04:54:58.572002 11401 consensus_queue.cc:260] T 9b1615195a4147019818c46ba67a9e4e P b80ee883e0fe4270a3a641a60c8dcc72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "438e234cf3d64a5890b7daad13a4fc40" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 34189 } } peers { permanent_uuid: "b80ee883e0fe4270a3a641a60c8dcc72" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 36591 } } peers { permanent_uuid: "a28b313a46aa459d822df7ecd20cb321" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 46029 } }
I20260925 04:54:58.573792 11401 ts_tablet_manager.cc:1428] T 9b1615195a4147019818c46ba67a9e4e P b80ee883e0fe4270a3a641a60c8dcc72: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260925 04:54:58.632547 11408 raft_consensus.cc:482] T 9b1615195a4147019818c46ba67a9e4e P b80ee883e0fe4270a3a641a60c8dcc72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:54:58.632907 11408 raft_consensus.cc:504] T 9b1615195a4147019818c46ba67a9e4e P b80ee883e0fe4270a3a641a60c8dcc72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "438e234cf3d64a5890b7daad13a4fc40" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 34189 } } peers { permanent_uuid: "b80ee883e0fe4270a3a641a60c8dcc72" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 36591 } } peers { permanent_uuid: "a28b313a46aa459d822df7ecd20cb321" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 46029 } }
I20260925 04:54:58.634708 11408 leader_election.cc:288] T 9b1615195a4147019818c46ba67a9e4e P b80ee883e0fe4270a3a641a60c8dcc72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 438e234cf3d64a5890b7daad13a4fc40 (127.6.29.131:34189), a28b313a46aa459d822df7ecd20cb321 (127.6.29.130:46029)
I20260925 04:54:58.638073 11396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:58.635114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:58485 (local address 127.6.29.131:34189)
0925 04:54:58.635334 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:58.635342 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:58.636050 (+ 708us) server_negotiation.cc:407] Connection header received
0925 04:54:58.636264 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:58.636269 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:58.636317 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:58.636419 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:58.637450 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:58.637456 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:58.637461 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:54:58.637478 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:58.637518 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:54:58.637529 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:58.637659 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:58.637740 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 04:54:58.637805 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260925 04:54:58.638182 11399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:58.635374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:56017 (local address 127.6.29.130:46029)
0925 04:54:58.635596 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:58.635604 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:58.636450 (+ 846us) server_negotiation.cc:407] Connection header received
0925 04:54:58.636626 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:58.636635 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:58.636699 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:58.636812 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:58.637567 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:58.637572 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:58.637576 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:54:58.637594 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:58.637630 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:54:58.637640 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:58.637762 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:58.637853 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 04:54:58.637917 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260925 04:54:58.638422 11409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:58.635083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.131:34189 (local address 127.6.29.129:58485)
0925 04:54:58.635927 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:58.635944 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:58.636098 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:58.636440 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:58.636446 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:58.636470 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:54:58.637212 (+ 742us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:54:58.637217 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:54:58.637238 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:54:58.637290 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:54:58.637629 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:54:58.637636 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:54:58.637642 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 04:54:58.637796 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 04:54:58.637843 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"thread_start_us":408,"threads_started":1}
I20260925 04:54:58.638785 11410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:58.635083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:46029 (local address 127.6.29.129:56017)
0925 04:54:58.636343 (+ 1260us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:58.636357 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:58.636484 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:58.636832 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:58.636837 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:58.636861 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:54:58.637378 (+ 517us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:54:58.637380 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:54:58.637392 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:54:58.637425 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:54:58.637743 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:54:58.637748 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:54:58.637753 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 04:54:58.637898 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 04:54:58.637942 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1141,"mutex_wait_us":386,"thread_start_us":426,"threads_started":1}
I20260925 04:54:58.639390 11337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b1615195a4147019818c46ba67a9e4e" candidate_uuid: "b80ee883e0fe4270a3a641a60c8dcc72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "438e234cf3d64a5890b7daad13a4fc40" is_pre_election: true
I20260925 04:54:58.639705 11207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b1615195a4147019818c46ba67a9e4e" candidate_uuid: "b80ee883e0fe4270a3a641a60c8dcc72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a28b313a46aa459d822df7ecd20cb321" is_pre_election: true
I20260925 04:54:58.639957 11337 raft_consensus.cc:2454] T 9b1615195a4147019818c46ba67a9e4e P 438e234cf3d64a5890b7daad13a4fc40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b80ee883e0fe4270a3a641a60c8dcc72 in term 0.
I20260925 04:54:58.640223 11207 raft_consensus.cc:2454] T 9b1615195a4147019818c46ba67a9e4e P a28b313a46aa459d822df7ecd20cb321 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b80ee883e0fe4270a3a641a60c8dcc72 in term 0.
I20260925 04:54:58.640669 11010 leader_election.cc:302] T 9b1615195a4147019818c46ba67a9e4e P b80ee883e0fe4270a3a641a60c8dcc72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 438e234cf3d64a5890b7daad13a4fc40, b80ee883e0fe4270a3a641a60c8dcc72; no voters:
I20260925 04:54:58.641060 11408 raft_consensus.cc:2781] T 9b1615195a4147019818c46ba67a9e4e P b80ee883e0fe4270a3a641a60c8dcc72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:54:58.641206 11408 raft_consensus.cc:482] T 9b1615195a4147019818c46ba67a9e4e P b80ee883e0fe4270a3a641a60c8dcc72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:54:58.641309 11408 raft_consensus.cc:3037] T 9b1615195a4147019818c46ba67a9e4e P b80ee883e0fe4270a3a641a60c8dcc72 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:58.643456 11408 raft_consensus.cc:504] T 9b1615195a4147019818c46ba67a9e4e P b80ee883e0fe4270a3a641a60c8dcc72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "438e234cf3d64a5890b7daad13a4fc40" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 34189 } } peers { permanent_uuid: "b80ee883e0fe4270a3a641a60c8dcc72" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 36591 } } peers { permanent_uuid: "a28b313a46aa459d822df7ecd20cb321" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 46029 } }
I20260925 04:54:58.644155 11408 leader_election.cc:288] T 9b1615195a4147019818c46ba67a9e4e P b80ee883e0fe4270a3a641a60c8dcc72 [CANDIDATE]: Term 1 election: Requested vote from peers 438e234cf3d64a5890b7daad13a4fc40 (127.6.29.131:34189), a28b313a46aa459d822df7ecd20cb321 (127.6.29.130:46029)
I20260925 04:54:58.644644 11337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b1615195a4147019818c46ba67a9e4e" candidate_uuid: "b80ee883e0fe4270a3a641a60c8dcc72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "438e234cf3d64a5890b7daad13a4fc40"
I20260925 04:54:58.644668 11207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b1615195a4147019818c46ba67a9e4e" candidate_uuid: "b80ee883e0fe4270a3a641a60c8dcc72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a28b313a46aa459d822df7ecd20cb321"
I20260925 04:54:58.644867 11337 raft_consensus.cc:3037] T 9b1615195a4147019818c46ba67a9e4e P 438e234cf3d64a5890b7daad13a4fc40 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:58.644863 11207 raft_consensus.cc:3037] T 9b1615195a4147019818c46ba67a9e4e P a28b313a46aa459d822df7ecd20cb321 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:58.648155 11207 raft_consensus.cc:2454] T 9b1615195a4147019818c46ba67a9e4e P a28b313a46aa459d822df7ecd20cb321 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b80ee883e0fe4270a3a641a60c8dcc72 in term 1.
I20260925 04:54:58.648145 11337 raft_consensus.cc:2454] T 9b1615195a4147019818c46ba67a9e4e P 438e234cf3d64a5890b7daad13a4fc40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b80ee883e0fe4270a3a641a60c8dcc72 in term 1.
I20260925 04:54:58.648567 11011 leader_election.cc:302] T 9b1615195a4147019818c46ba67a9e4e P b80ee883e0fe4270a3a641a60c8dcc72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a28b313a46aa459d822df7ecd20cb321, b80ee883e0fe4270a3a641a60c8dcc72; no voters:
I20260925 04:54:58.648860 11408 raft_consensus.cc:2781] T 9b1615195a4147019818c46ba67a9e4e P b80ee883e0fe4270a3a641a60c8dcc72 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:54:58.649281 11408 raft_consensus.cc:686] T 9b1615195a4147019818c46ba67a9e4e P b80ee883e0fe4270a3a641a60c8dcc72 [term 1 LEADER]: Becoming Leader. State: Replica: b80ee883e0fe4270a3a641a60c8dcc72, State: Running, Role: LEADER
I20260925 04:54:58.649883 11408 consensus_queue.cc:237] T 9b1615195a4147019818c46ba67a9e4e P b80ee883e0fe4270a3a641a60c8dcc72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "438e234cf3d64a5890b7daad13a4fc40" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 34189 } } peers { permanent_uuid: "b80ee883e0fe4270a3a641a60c8dcc72" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 36591 } } peers { permanent_uuid: "a28b313a46aa459d822df7ecd20cb321" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 46029 } }
I20260925 04:54:58.654431 10934 catalog_manager.cc:5495] T 9b1615195a4147019818c46ba67a9e4e P b80ee883e0fe4270a3a641a60c8dcc72 reported cstate change: term changed from 0 to 1, leader changed from <none> to b80ee883e0fe4270a3a641a60c8dcc72 (127.6.29.129). New cstate: current_term: 1 leader_uuid: "b80ee883e0fe4270a3a641a60c8dcc72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "438e234cf3d64a5890b7daad13a4fc40" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 34189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b80ee883e0fe4270a3a641a60c8dcc72" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 36591 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a28b313a46aa459d822df7ecd20cb321" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 46029 } health_report { overall_health: UNKNOWN } } }
W20260925 04:54:58.702656 11383 tablet.cc:2369] T 9b1615195a4147019818c46ba67a9e4e P 438e234cf3d64a5890b7daad13a4fc40: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:54:58.703287 11397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:58.701164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43036 (local address 127.6.29.129:36591)
0925 04:54:58.701342 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:58.701347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:58.701363 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:54:58.701452 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:58.701456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:58.701534 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:58.701616 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:58.702601 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:58.702605 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:58.702608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:54:58.702620 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:58.702647 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:54:58.702658 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:58.702765 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:58.702942 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 04:54:58.703004 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 04:54:58.711705 11337 raft_consensus.cc:1261] T 9b1615195a4147019818c46ba67a9e4e P 438e234cf3d64a5890b7daad13a4fc40 [term 1 FOLLOWER]: Refusing update from remote peer b80ee883e0fe4270a3a641a60c8dcc72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:54:58.711740 11207 raft_consensus.cc:1261] T 9b1615195a4147019818c46ba67a9e4e P a28b313a46aa459d822df7ecd20cb321 [term 1 FOLLOWER]: Refusing update from remote peer b80ee883e0fe4270a3a641a60c8dcc72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:54:58.712716 11411 consensus_queue.cc:1035] T 9b1615195a4147019818c46ba67a9e4e P b80ee883e0fe4270a3a641a60c8dcc72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a28b313a46aa459d822df7ecd20cb321" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 46029 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:54:58.713085 11408 consensus_queue.cc:1035] T 9b1615195a4147019818c46ba67a9e4e P b80ee883e0fe4270a3a641a60c8dcc72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "438e234cf3d64a5890b7daad13a4fc40" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 34189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:54:58.738402 10934 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:54:58.738699 10934 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37192:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:54:58.742647 10934 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 19489055a8934759b9fc934c52199e2c: Sending DeleteTablet for 3 replicas of tablet 9b1615195a4147019818c46ba67a9e4e
I20260925 04:54:58.743894 11187 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b1615195a4147019818c46ba67a9e4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:54:58 UTC) from {username='slave'} at 127.0.0.1:46526
I20260925 04:54:58.743896 11317 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b1615195a4147019818c46ba67a9e4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:54:58 UTC) from {username='slave'} at 127.0.0.1:45264
I20260925 04:54:58.743894 11056 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b1615195a4147019818c46ba67a9e4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:54:58 UTC) from {username='slave'} at 127.0.0.1:43034
I20260925 04:54:58.745234 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 10994
I20260925 04:54:58.746542 11425 tablet_replica.cc:330] T 9b1615195a4147019818c46ba67a9e4e P 438e234cf3d64a5890b7daad13a4fc40: stopping tablet replica
I20260925 04:54:58.747057 11425 raft_consensus.cc:2229] T 9b1615195a4147019818c46ba67a9e4e P 438e234cf3d64a5890b7daad13a4fc40 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:54:58.747628 11425 raft_consensus.cc:2258] T 9b1615195a4147019818c46ba67a9e4e P 438e234cf3d64a5890b7daad13a4fc40 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:54:58.750424 11425 ts_tablet_manager.cc:1905] T 9b1615195a4147019818c46ba67a9e4e P 438e234cf3d64a5890b7daad13a4fc40: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:54:58.759631 11425 ts_tablet_manager.cc:1918] T 9b1615195a4147019818c46ba67a9e4e P 438e234cf3d64a5890b7daad13a4fc40: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:54:58.760254 11425 log.cc:1196] T 9b1615195a4147019818c46ba67a9e4e P 438e234cf3d64a5890b7daad13a4fc40: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-2/wal/wals/9b1615195a4147019818c46ba67a9e4e
I20260925 04:54:58.760890 11425 ts_tablet_manager.cc:1939] T 9b1615195a4147019818c46ba67a9e4e P 438e234cf3d64a5890b7daad13a4fc40: Deleting consensus metadata
I20260925 04:54:58.761401 11397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:58.747948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43046 (local address 127.6.29.129:36591)
0925 04:54:58.748169 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:58.748178 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:58.758968 (+ 10790us) server_negotiation.cc:407] Connection header received
0925 04:54:58.759244 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:58.759254 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:58.759346 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:58.759478 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:58.760534 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:58.760541 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:58.760546 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:54:58.760568 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:58.760607 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:54:58.760625 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:58.760731 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:58.761008 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 04:54:58.761074 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 04:54:58.762383 11116 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:54:58.762382 11423 tablet_replica.cc:330] T 9b1615195a4147019818c46ba67a9e4e P a28b313a46aa459d822df7ecd20cb321: stopping tablet replica
I20260925 04:54:58.762938 11423 raft_consensus.cc:2229] T 9b1615195a4147019818c46ba67a9e4e P a28b313a46aa459d822df7ecd20cb321 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:54:58.763109 10920 catalog_manager.cc:4841] TS 438e234cf3d64a5890b7daad13a4fc40 (127.6.29.131:34189): tablet 9b1615195a4147019818c46ba67a9e4e (table test-table [id=b884da746c294da7a89965e914d08bbd]) successfully deleted
I20260925 04:54:58.763564 11423 raft_consensus.cc:2258] T 9b1615195a4147019818c46ba67a9e4e P a28b313a46aa459d822df7ecd20cb321 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:54:58.765724 11423 ts_tablet_manager.cc:1905] T 9b1615195a4147019818c46ba67a9e4e P a28b313a46aa459d822df7ecd20cb321: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:54:58.773370 11423 ts_tablet_manager.cc:1918] T 9b1615195a4147019818c46ba67a9e4e P a28b313a46aa459d822df7ecd20cb321: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:54:58.773496 11423 log.cc:1196] T 9b1615195a4147019818c46ba67a9e4e P a28b313a46aa459d822df7ecd20cb321: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790312053666503-6262-0/minicluster-data/ts-1/wal/wals/9b1615195a4147019818c46ba67a9e4e
I20260925 04:54:58.774003 11423 ts_tablet_manager.cc:1939] T 9b1615195a4147019818c46ba67a9e4e P a28b313a46aa459d822df7ecd20cb321: Deleting consensus metadata
I20260925 04:54:58.775508 10921 catalog_manager.cc:4841] TS a28b313a46aa459d822df7ecd20cb321 (127.6.29.130:46029): tablet 9b1615195a4147019818c46ba67a9e4e (table test-table [id=b884da746c294da7a89965e914d08bbd]) successfully deleted
I20260925 04:54:58.862088 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 10994
W20260925 04:54:58.879117 10919 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.29.129:36591 (error 108)
I20260925 04:54:58.879868 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 11125
I20260925 04:54:58.881095 11398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:58.880591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:36591 (local address 127.0.0.1:43048)
0925 04:54:58.880787 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:58.880918 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.129:36591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 04:54:58.881347 10919 catalog_manager.cc:4584] TS b80ee883e0fe4270a3a641a60c8dcc72 (127.6.29.129:36591): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9b1615195a4147019818c46ba67a9e4e: Network error: Client connection negotiation failed: client connection to 127.6.29.129:36591: connect: Connection refused (error 111)
I20260925 04:54:58.882836 11399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:58.880655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46538 (local address 127.6.29.130:46029)
0925 04:54:58.880854 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:58.880859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:58.881007 (+ 148us) server_negotiation.cc:407] Connection header received
0925 04:54:58.881206 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:58.881215 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:58.881328 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:58.881445 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:58.882254 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:58.882259 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:58.882262 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:54:58.882276 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:58.882303 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:54:58.882314 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:58.882414 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:58.882476 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 04:54:58.882518 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 04:54:58.883430 11247 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:54:58.935035 11398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:58.934492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:36591 (local address 127.0.0.1:43054)
0925 04:54:58.934737 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:58.934843 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.129:36591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260925 04:54:58.936162 11398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:58.935774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:36591 (local address 127.0.0.1:43056)
0925 04:54:58.935939 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:58.936030 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.129:36591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 04:54:58.973151 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 11125
I20260925 04:54:58.989470 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 11255
I20260925 04:54:58.992455 11396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:58.990278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45274 (local address 127.6.29.131:34189)
0925 04:54:58.990456 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:58.990464 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:58.990489 (+ 25us) server_negotiation.cc:407] Connection header received
0925 04:54:58.990700 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:58.990706 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:58.990802 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:58.990914 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:58.991794 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:58.991800 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:58.991805 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:54:58.991823 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:58.991867 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:54:58.991879 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:58.992014 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:58.992107 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 04:54:58.992167 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 04:54:58.993173 11377 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:54:58.997442 11398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:58.996850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:36591 (local address 127.0.0.1:43064)
0925 04:54:58.997141 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:58.997244 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.129:36591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260925 04:54:58.998420 11398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:58.998102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:36591 (local address 127.0.0.1:43078)
0925 04:54:58.998244 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:58.998310 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.129:36591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 04:54:59.079429 11398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:59.078852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:36591 (local address 127.0.0.1:43086)
0925 04:54:59.079163 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:59.079286 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.129:36591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
I20260925 04:54:59.080667 11398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:59.080299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:36591 (local address 127.0.0.1:43092)
0925 04:54:59.080487 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:59.080553 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.129:36591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260925 04:54:59.082506 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 11255
I20260925 04:54:59.100020 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 10904
I20260925 04:54:59.101574 10965 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:54:59.206904 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 10904
2026-09-25T04:54:59Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2718 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 04:54:59.248979 6262 mini_oidc.cc:152] Starting JWKS server
I20260925 04:54:59.251577 6262 webserver.cc:460] Webserver started at https://127.0.0.1:39255/ using document root <none> and password file <none>
I20260925 04:54:59.251971 6262 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 04:54:59.252668 6262 webserver.cc:460] Webserver started at http://127.0.0.1:40227/ using document root <none> and password file <none>
2026-09-25T04:54:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:54:59Z Disabled control of system clock
I20260925 04:54:59.280324 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:39505
--webserver_interface=127.6.29.190
--webserver_port=0
--builtin_ntp_servers=127.6.29.148:40951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:39505
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39255/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312053666503-6262-0/testchainca.cert with env {}
W20260925 04:54:59.613883 11437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:54:59.614193 11437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:54:59.614280 11437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:54:59.621639 11437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:54:59.621814 11437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:54:59.621866 11437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:54:59.621924 11437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:54:59.621968 11437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 04:54:59.622022 11437 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 04:54:59.622061 11437 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39255/jwks.json
I20260925 04:54:59.632285 11437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:40951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:39505
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:39505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39255/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312053666503-6262-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:54:59.633571 11437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:54:59.634862 11437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:54:59.647461 11443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:59.647408 11442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:54:59.647511 11445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:54:59.648564 11437 server_base.cc:1034] running on GCE node
I20260925 04:54:59.649448 11437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:54:59.650969 11437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:54:59.652199 11437 hybrid_clock.cc:648] HybridClock initialized: now 1790312099652159 us; error 40 us; skew 500 ppm
I20260925 04:54:59.652567 11437 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 04:54:59.654927 11437 webserver.cc:460] Webserver started at http://127.6.29.190:34657/ using document root <none> and password file <none>
I20260925 04:54:59.655640 11437 fs_manager.cc:359] Metadata directory not provided
I20260925 04:54:59.655823 11437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:54:59.656147 11437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:54:59.659200 11437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "d78c249229c24fdc8aaabd002f2aecd0"
format_stamp: "Formatted at 2026-09-25 04:54:59 on dist-test-slave-xzjr"
server_key: "58cdd626b0049e0d031fb9843455e211"
server_key_iv: "626946d929838bed53625e6326484917"
server_key_version: "encryptionkey@0"
I20260925 04:54:59.660408 11437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "d78c249229c24fdc8aaabd002f2aecd0"
format_stamp: "Formatted at 2026-09-25 04:54:59 on dist-test-slave-xzjr"
server_key: "58cdd626b0049e0d031fb9843455e211"
server_key_iv: "626946d929838bed53625e6326484917"
server_key_version: "encryptionkey@0"
I20260925 04:54:59.666078 11437 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260925 04:54:59.669893 11451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:54:59.671523 11437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 04:54:59.671813 11437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "d78c249229c24fdc8aaabd002f2aecd0"
format_stamp: "Formatted at 2026-09-25 04:54:59 on dist-test-slave-xzjr"
server_key: "58cdd626b0049e0d031fb9843455e211"
server_key_iv: "626946d929838bed53625e6326484917"
server_key_version: "encryptionkey@0"
I20260925 04:54:59.672076 11437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:54:59.702306 11437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:54:59.703263 11437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:54:59.703603 11437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:54:59.722633 11437 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:39505
I20260925 04:54:59.722643 11502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:39505 every 8 connection(s)
I20260925 04:54:59.724076 11437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:54:59.727942 11503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:54:59.730425 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 11437
I20260925 04:54:59.730630 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 04:54:59.731256 6262 external_mini_cluster.cc:1428] Setting key 72e7fc0c9a2eb427293593ae1e7fc83b
I20260925 04:54:59.741478 11503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d78c249229c24fdc8aaabd002f2aecd0: Bootstrap starting.
I20260925 04:54:59.745649 11503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d78c249229c24fdc8aaabd002f2aecd0: Neither blocks nor log segments found. Creating new log.
I20260925 04:54:59.746968 11503 log.cc:824] T 00000000000000000000000000000000 P d78c249229c24fdc8aaabd002f2aecd0: Log is configured to *not* fsync() on all Append() calls
I20260925 04:54:59.747421 11506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:59.733306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39064 (local address 127.6.29.190:39505)
0925 04:54:59.734345 (+ 1039us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:59.734359 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:59.734411 (+ 52us) server_negotiation.cc:407] Connection header received
0925 04:54:59.736841 (+ 2430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:59.736879 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:59.737548 (+ 669us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:59.738049 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:54:59.738644 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:59.740610 (+ 1966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:59.742082 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:59.743091 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:59.743327 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:59.744507 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:54:59.744517 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:54:59.744535 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:54:59.745275 (+ 740us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:54:59.745401 (+ 126us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:54:59.745430 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:54:59.745568 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 04:54:59.745733 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 04:54:59.746103 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":299,"threads_started":1}
I20260925 04:54:59.750065 11503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d78c249229c24fdc8aaabd002f2aecd0: No bootstrap required, opened a new log
I20260925 04:54:59.751255 6262 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:54:59.757577 11503 raft_consensus.cc:348] T 00000000000000000000000000000000 P d78c249229c24fdc8aaabd002f2aecd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d78c249229c24fdc8aaabd002f2aecd0" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39505 } }
I20260925 04:54:59.757942 11503 raft_consensus.cc:374] T 00000000000000000000000000000000 P d78c249229c24fdc8aaabd002f2aecd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:54:59.758066 11503 raft_consensus.cc:729] T 00000000000000000000000000000000 P d78c249229c24fdc8aaabd002f2aecd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d78c249229c24fdc8aaabd002f2aecd0, State: Initialized, Role: FOLLOWER
I20260925 04:54:59.758709 11503 consensus_queue.cc:260] T 00000000000000000000000000000000 P d78c249229c24fdc8aaabd002f2aecd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d78c249229c24fdc8aaabd002f2aecd0" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39505 } }
I20260925 04:54:59.758970 11503 raft_consensus.cc:388] T 00000000000000000000000000000000 P d78c249229c24fdc8aaabd002f2aecd0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:54:59.759119 11503 raft_consensus.cc:482] T 00000000000000000000000000000000 P d78c249229c24fdc8aaabd002f2aecd0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:54:59.759349 11503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d78c249229c24fdc8aaabd002f2aecd0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:54:59.761576 11503 raft_consensus.cc:504] T 00000000000000000000000000000000 P d78c249229c24fdc8aaabd002f2aecd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d78c249229c24fdc8aaabd002f2aecd0" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39505 } }
I20260925 04:54:59.762107 11503 leader_election.cc:302] T 00000000000000000000000000000000 P d78c249229c24fdc8aaabd002f2aecd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d78c249229c24fdc8aaabd002f2aecd0; no voters:
I20260925 04:54:59.762647 11503 leader_election.cc:288] T 00000000000000000000000000000000 P d78c249229c24fdc8aaabd002f2aecd0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:54:59.762797 11508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d78c249229c24fdc8aaabd002f2aecd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:54:59.763947 11508 raft_consensus.cc:686] T 00000000000000000000000000000000 P d78c249229c24fdc8aaabd002f2aecd0 [term 1 LEADER]: Becoming Leader. State: Replica: d78c249229c24fdc8aaabd002f2aecd0, State: Running, Role: LEADER
I20260925 04:54:59.764451 11508 consensus_queue.cc:237] T 00000000000000000000000000000000 P d78c249229c24fdc8aaabd002f2aecd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d78c249229c24fdc8aaabd002f2aecd0" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39505 } }
I20260925 04:54:59.765097 11503 sys_catalog.cc:564] T 00000000000000000000000000000000 P d78c249229c24fdc8aaabd002f2aecd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:54:59.767797 11509 sys_catalog.cc:455] T 00000000000000000000000000000000 P d78c249229c24fdc8aaabd002f2aecd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d78c249229c24fdc8aaabd002f2aecd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d78c249229c24fdc8aaabd002f2aecd0" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39505 } } }
I20260925 04:54:59.767869 11510 sys_catalog.cc:455] T 00000000000000000000000000000000 P d78c249229c24fdc8aaabd002f2aecd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d78c249229c24fdc8aaabd002f2aecd0. Latest consensus state: current_term: 1 leader_uuid: "d78c249229c24fdc8aaabd002f2aecd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d78c249229c24fdc8aaabd002f2aecd0" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39505 } } }
I20260925 04:54:59.768329 11510 sys_catalog.cc:458] T 00000000000000000000000000000000 P d78c249229c24fdc8aaabd002f2aecd0 [sys.catalog]: This master's current role is: LEADER
I20260925 04:54:59.768334 11509 sys_catalog.cc:458] T 00000000000000000000000000000000 P d78c249229c24fdc8aaabd002f2aecd0 [sys.catalog]: This master's current role is: LEADER
I20260925 04:54:59.769196 11517 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:54:59.775029 11517 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:54:59.784047 11517 catalog_manager.cc:1347] Generated new cluster ID: 8441bfbd2fe14b51ada2a1baabcf1162
I20260925 04:54:59.784180 11517 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:54:59.814949 11517 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:54:59.816349 11517 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:54:59.824988 11517 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d78c249229c24fdc8aaabd002f2aecd0: Generated new TSK 0
I20260925 04:54:59.826109 11517 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 04:54:59.870431 11533 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 04:54:59.870643 11533 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 04:54:59.871618 11506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:54:59.849450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39080 (local address 127.6.29.190:39505)
0925 04:54:59.849684 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 04:54:59.849691 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:54:59.850336 (+ 645us) server_negotiation.cc:407] Connection header received
0925 04:54:59.850692 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:54:59.850696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:54:59.850780 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:54:59.850906 (+ 126us) server_negotiation.cc:226] Negotiated authn=JWT
0925 04:54:59.851569 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:59.852407 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:59.854635 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:54:59.855114 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:54:59.855218 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:59.855913 (+ 695us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 04:54:59.871074 (+ 15161us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39255/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 04:54:59.871391 (+ 317us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39080: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39255/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":67}
W20260925 04:54:59.871795 11506 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39080: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39255/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 04:54:59.873733 11532 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:54:59.849439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39505 (local address 127.0.0.1:39080)
0925 04:54:59.850200 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0925 04:54:59.850222 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 04:54:59.850417 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:54:59.850965 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:54:59.850971 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:54:59.850984 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0925 04:54:59.851383 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:59.851396 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:59.852547 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:59.852554 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:59.854457 (+ 1903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:54:59.854472 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:54:59.855303 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:54:59.855310 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:54:59.855496 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:54:59.855576 (+ 80us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 04:54:59.871423 (+ 15847us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39255/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 04:54:59.871855 (+ 432us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.29.190:39505: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39255/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":399,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.6.29.190:39505: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39255/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 04:54:59.875684 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 11437
I20260925 04:54:59.877024 11498 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:54:59.973240 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 11437
2026-09-25T04:55:00Z chronyd exiting
I20260925 04:55:00.020618 6262 test_util.cc:175] -----------------------------------------------
I20260925 04:55:00.020748 6262 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790312053666503-6262-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (776 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 04:55:00.024576 6262 mini_oidc.cc:152] Starting JWKS server
I20260925 04:55:00.026448 6262 webserver.cc:460] Webserver started at https://127.0.0.1:45137/ using document root <none> and password file <none>
I20260925 04:55:00.026716 6262 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 04:55:00.027450 6262 webserver.cc:460] Webserver started at http://127.0.0.1:39981/ using document root <none> and password file <none>
2026-09-25T04:55:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:55:00Z Disabled control of system clock
I20260925 04:55:00.054122 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:35649
--webserver_interface=127.6.29.190
--webserver_port=0
--builtin_ntp_servers=127.6.29.148:35061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:35649
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45137/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312053666503-6262-0/testchainca.cert with env {}
W20260925 04:55:00.389019 11540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:55:00.389394 11540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:55:00.389554 11540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:55:00.397017 11540 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:55:00.397184 11540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:55:00.397298 11540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:55:00.397413 11540 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:55:00.397490 11540 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 04:55:00.397579 11540 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 04:55:00.397655 11540 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45137/jwks.json
I20260925 04:55:00.407840 11540 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:35061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:35649
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:35649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45137/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312053666503-6262-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:55:00.409266 11540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:55:00.410739 11540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:55:00.422773 11545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:00.422817 11546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:00.422880 11548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:55:00.424078 11540 server_base.cc:1034] running on GCE node
I20260925 04:55:00.424940 11540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:55:00.426427 11540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:55:00.427647 11540 hybrid_clock.cc:648] HybridClock initialized: now 1790312100427595 us; error 53 us; skew 500 ppm
I20260925 04:55:00.428040 11540 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 04:55:00.430346 11540 webserver.cc:460] Webserver started at http://127.6.29.190:42201/ using document root <none> and password file <none>
I20260925 04:55:00.431048 11540 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:00.431205 11540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:00.431538 11540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:55:00.434571 11540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "c15440d896be4b21868d3ceff9cf1a26"
format_stamp: "Formatted at 2026-09-25 04:55:00 on dist-test-slave-xzjr"
server_key: "26e19ae4b006be048dee0832e29ff37b"
server_key_iv: "c26a0d0b64431d886ce296885e59fd76"
server_key_version: "encryptionkey@0"
I20260925 04:55:00.435388 11540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "c15440d896be4b21868d3ceff9cf1a26"
format_stamp: "Formatted at 2026-09-25 04:55:00 on dist-test-slave-xzjr"
server_key: "26e19ae4b006be048dee0832e29ff37b"
server_key_iv: "c26a0d0b64431d886ce296885e59fd76"
server_key_version: "encryptionkey@0"
I20260925 04:55:00.440791 11540 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260925 04:55:00.444890 11554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:00.446478 11540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 04:55:00.446705 11540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "c15440d896be4b21868d3ceff9cf1a26"
format_stamp: "Formatted at 2026-09-25 04:55:00 on dist-test-slave-xzjr"
server_key: "26e19ae4b006be048dee0832e29ff37b"
server_key_iv: "c26a0d0b64431d886ce296885e59fd76"
server_key_version: "encryptionkey@0"
I20260925 04:55:00.446967 11540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:00.475116 11540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:55:00.476176 11540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:55:00.476508 11540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:55:00.495633 11540 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:35649
I20260925 04:55:00.495652 11605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:35649 every 8 connection(s)
I20260925 04:55:00.497114 11540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:55:00.501122 11606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:55:00.504011 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 11540
I20260925 04:55:00.504232 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 04:55:00.504855 6262 external_mini_cluster.cc:1428] Setting key 0ccbb0ce9a2c942ea7c42218c8b5d951
I20260925 04:55:00.513348 11606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c15440d896be4b21868d3ceff9cf1a26: Bootstrap starting.
I20260925 04:55:00.516716 11606 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c15440d896be4b21868d3ceff9cf1a26: Neither blocks nor log segments found. Creating new log.
I20260925 04:55:00.517887 11606 log.cc:824] T 00000000000000000000000000000000 P c15440d896be4b21868d3ceff9cf1a26: Log is configured to *not* fsync() on all Append() calls
I20260925 04:55:00.518878 11609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:00.506931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54224 (local address 127.6.29.190:35649)
0925 04:55:00.508080 (+ 1149us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:00.508092 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:00.508146 (+ 54us) server_negotiation.cc:407] Connection header received
0925 04:55:00.509800 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:00.509824 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:00.510343 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:00.510859 (+ 516us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:00.511465 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:00.512770 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:00.514130 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:00.514815 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:00.515008 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:00.516239 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:00.516246 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:00.516261 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:55:00.516804 (+ 543us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:00.516947 (+ 143us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:55:00.516969 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:00.517102 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:00.517337 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 04:55:00.517638 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":356,"threads_started":1}
I20260925 04:55:00.521274 11606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c15440d896be4b21868d3ceff9cf1a26: No bootstrap required, opened a new log
I20260925 04:55:00.523238 6262 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:55:00.529364 11606 raft_consensus.cc:348] T 00000000000000000000000000000000 P c15440d896be4b21868d3ceff9cf1a26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15440d896be4b21868d3ceff9cf1a26" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 35649 } }
I20260925 04:55:00.529779 11606 raft_consensus.cc:374] T 00000000000000000000000000000000 P c15440d896be4b21868d3ceff9cf1a26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:55:00.529894 11606 raft_consensus.cc:729] T 00000000000000000000000000000000 P c15440d896be4b21868d3ceff9cf1a26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c15440d896be4b21868d3ceff9cf1a26, State: Initialized, Role: FOLLOWER
I20260925 04:55:00.530526 11606 consensus_queue.cc:260] T 00000000000000000000000000000000 P c15440d896be4b21868d3ceff9cf1a26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15440d896be4b21868d3ceff9cf1a26" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 35649 } }
I20260925 04:55:00.530938 11606 raft_consensus.cc:388] T 00000000000000000000000000000000 P c15440d896be4b21868d3ceff9cf1a26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:55:00.531100 11606 raft_consensus.cc:482] T 00000000000000000000000000000000 P c15440d896be4b21868d3ceff9cf1a26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:55:00.531278 11606 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c15440d896be4b21868d3ceff9cf1a26 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:55:00.533643 11606 raft_consensus.cc:504] T 00000000000000000000000000000000 P c15440d896be4b21868d3ceff9cf1a26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15440d896be4b21868d3ceff9cf1a26" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 35649 } }
I20260925 04:55:00.534165 11606 leader_election.cc:302] T 00000000000000000000000000000000 P c15440d896be4b21868d3ceff9cf1a26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c15440d896be4b21868d3ceff9cf1a26; no voters:
I20260925 04:55:00.534863 11606 leader_election.cc:288] T 00000000000000000000000000000000 P c15440d896be4b21868d3ceff9cf1a26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:55:00.535000 11611 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c15440d896be4b21868d3ceff9cf1a26 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:55:00.535717 11611 raft_consensus.cc:686] T 00000000000000000000000000000000 P c15440d896be4b21868d3ceff9cf1a26 [term 1 LEADER]: Becoming Leader. State: Replica: c15440d896be4b21868d3ceff9cf1a26, State: Running, Role: LEADER
I20260925 04:55:00.536324 11611 consensus_queue.cc:237] T 00000000000000000000000000000000 P c15440d896be4b21868d3ceff9cf1a26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15440d896be4b21868d3ceff9cf1a26" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 35649 } }
I20260925 04:55:00.537289 11606 sys_catalog.cc:564] T 00000000000000000000000000000000 P c15440d896be4b21868d3ceff9cf1a26 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:55:00.539206 11612 sys_catalog.cc:455] T 00000000000000000000000000000000 P c15440d896be4b21868d3ceff9cf1a26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c15440d896be4b21868d3ceff9cf1a26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15440d896be4b21868d3ceff9cf1a26" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 35649 } } }
I20260925 04:55:00.539597 11612 sys_catalog.cc:458] T 00000000000000000000000000000000 P c15440d896be4b21868d3ceff9cf1a26 [sys.catalog]: This master's current role is: LEADER
I20260925 04:55:00.539932 11613 sys_catalog.cc:455] T 00000000000000000000000000000000 P c15440d896be4b21868d3ceff9cf1a26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c15440d896be4b21868d3ceff9cf1a26. Latest consensus state: current_term: 1 leader_uuid: "c15440d896be4b21868d3ceff9cf1a26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c15440d896be4b21868d3ceff9cf1a26" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 35649 } } }
I20260925 04:55:00.540210 11613 sys_catalog.cc:458] T 00000000000000000000000000000000 P c15440d896be4b21868d3ceff9cf1a26 [sys.catalog]: This master's current role is: LEADER
I20260925 04:55:00.548297 11618 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:55:00.554000 11618 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:55:00.562067 11618 catalog_manager.cc:1347] Generated new cluster ID: 506bd4b824d34a5fb942ccf46a66e691
I20260925 04:55:00.562196 11618 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:55:00.592825 11618 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:55:00.594372 11618 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:55:00.606372 11618 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c15440d896be4b21868d3ceff9cf1a26: Generated new TSK 0
I20260925 04:55:00.607326 11618 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 04:55:00.660861 11636 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 04:55:00.661005 11636 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 04:55:00.662014 11609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:00.633927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54232 (local address 127.6.29.190:35649)
0925 04:55:00.634143 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:00.634150 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:00.634498 (+ 348us) server_negotiation.cc:407] Connection header received
0925 04:55:00.634825 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:00.634830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:00.634902 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:00.635045 (+ 143us) server_negotiation.cc:226] Negotiated authn=JWT
0925 04:55:00.635591 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:00.636437 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:00.638557 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:00.639126 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:00.639237 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:00.639919 (+ 682us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 04:55:00.661451 (+ 21532us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45137/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 04:55:00.661785 (+ 334us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54232: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45137/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 04:55:00.662177 11609 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54232: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45137/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 04:55:00.662429 11635 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:00.633742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:35649 (local address 127.0.0.1:54232)
0925 04:55:00.634379 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:00.634400 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:00.634558 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:00.635089 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:00.635094 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:00.635107 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0925 04:55:00.635404 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:00.635417 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:00.636598 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:00.636605 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:00.638341 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:00.638359 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:00.639303 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:00.639310 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:00.639509 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:00.639557 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 04:55:00.661725 (+ 22168us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45137/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 04:55:00.662023 (+ 298us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.29.190:35649: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45137/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":342,"threads_started":1}
I20260925 04:55:00.663484 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 11540
I20260925 04:55:00.664794 11601 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:55:00.762368 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 11540
2026-09-25T04:55:00Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (793 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 04:55:00.817973 6262 mini_oidc.cc:152] Starting JWKS server
I20260925 04:55:00.819904 6262 webserver.cc:460] Webserver started at https://127.0.0.1:34009/ using document root <none> and password file <none>
I20260925 04:55:00.820156 6262 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 04:55:00.820833 6262 webserver.cc:460] Webserver started at http://127.0.0.1:35655/ using document root <none> and password file <none>
2026-09-25T04:55:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:55:00Z Disabled control of system clock
I20260925 04:55:00.849035 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:40519
--webserver_interface=127.6.29.190
--webserver_port=0
--builtin_ntp_servers=127.6.29.148:36221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:40519
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34009/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 04:55:01.180368 11643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:55:01.180675 11643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:55:01.180763 11643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:55:01.188004 11643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:55:01.188110 11643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:55:01.188145 11643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:55:01.188201 11643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:55:01.188244 11643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 04:55:01.188297 11643 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 04:55:01.188338 11643 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34009/jwks.json
I20260925 04:55:01.197705 11643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:36221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:40519
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:40519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34009/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:55:01.199178 11643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:55:01.201179 11643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:55:01.213523 11649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:01.213610 11648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:01.213588 11651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:55:01.214148 11643 server_base.cc:1034] running on GCE node
I20260925 04:55:01.215085 11643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:55:01.216739 11643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:55:01.217978 11643 hybrid_clock.cc:648] HybridClock initialized: now 1790312101217931 us; error 49 us; skew 500 ppm
I20260925 04:55:01.218357 11643 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 04:55:01.220829 11643 webserver.cc:460] Webserver started at http://127.6.29.190:33371/ using document root <none> and password file <none>
I20260925 04:55:01.221562 11643 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:01.221698 11643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:01.222034 11643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:55:01.225129 11643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "86b680d6b9544f52b2f6526f95c66b74"
format_stamp: "Formatted at 2026-09-25 04:55:01 on dist-test-slave-xzjr"
server_key: "d1121bb9aca8fdf230b38c9bd19fbd27"
server_key_iv: "4f8e34978e99ac16fe590395f0429d84"
server_key_version: "encryptionkey@0"
I20260925 04:55:01.225962 11643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "86b680d6b9544f52b2f6526f95c66b74"
format_stamp: "Formatted at 2026-09-25 04:55:01 on dist-test-slave-xzjr"
server_key: "d1121bb9aca8fdf230b38c9bd19fbd27"
server_key_iv: "4f8e34978e99ac16fe590395f0429d84"
server_key_version: "encryptionkey@0"
I20260925 04:55:01.231379 11643 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 04:55:01.235023 11657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:01.236534 11643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:55:01.236737 11643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "86b680d6b9544f52b2f6526f95c66b74"
format_stamp: "Formatted at 2026-09-25 04:55:01 on dist-test-slave-xzjr"
server_key: "d1121bb9aca8fdf230b38c9bd19fbd27"
server_key_iv: "4f8e34978e99ac16fe590395f0429d84"
server_key_version: "encryptionkey@0"
I20260925 04:55:01.236964 11643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:01.248482 11643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:55:01.249531 11643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:55:01.249873 11643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:55:01.268528 11643 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:40519
I20260925 04:55:01.268546 11708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:40519 every 8 connection(s)
I20260925 04:55:01.269995 11643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:55:01.274088 11709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:55:01.278756 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 11643
I20260925 04:55:01.278959 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 04:55:01.279616 6262 external_mini_cluster.cc:1428] Setting key fb3831938682d7d81a99a6b1fbb5970d
I20260925 04:55:01.286299 11709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86b680d6b9544f52b2f6526f95c66b74: Bootstrap starting.
I20260925 04:55:01.289984 11709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86b680d6b9544f52b2f6526f95c66b74: Neither blocks nor log segments found. Creating new log.
I20260925 04:55:01.291268 11709 log.cc:824] T 00000000000000000000000000000000 P 86b680d6b9544f52b2f6526f95c66b74: Log is configured to *not* fsync() on all Append() calls
I20260925 04:55:01.294898 11709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86b680d6b9544f52b2f6526f95c66b74: No bootstrap required, opened a new log
I20260925 04:55:01.296061 11712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:01.281669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32910 (local address 127.6.29.190:40519)
0925 04:55:01.282798 (+ 1129us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:01.282818 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:01.282874 (+ 56us) server_negotiation.cc:407] Connection header received
0925 04:55:01.285207 (+ 2333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:01.285238 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:01.285845 (+ 607us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:01.286295 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:01.286866 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:01.288916 (+ 2050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:01.290344 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:01.291399 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:01.291568 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:01.293225 (+ 1657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:01.293235 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:01.293254 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:55:01.293952 (+ 698us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:01.294054 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:55:01.294078 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:01.294199 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:01.294408 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 04:55:01.294735 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":366,"threads_started":1}
I20260925 04:55:01.300252 6262 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:55:01.303627 11709 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86b680d6b9544f52b2f6526f95c66b74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b680d6b9544f52b2f6526f95c66b74" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40519 } }
I20260925 04:55:01.304054 11709 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86b680d6b9544f52b2f6526f95c66b74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:55:01.304180 11709 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86b680d6b9544f52b2f6526f95c66b74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86b680d6b9544f52b2f6526f95c66b74, State: Initialized, Role: FOLLOWER
I20260925 04:55:01.304739 11709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86b680d6b9544f52b2f6526f95c66b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b680d6b9544f52b2f6526f95c66b74" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40519 } }
I20260925 04:55:01.304965 11709 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86b680d6b9544f52b2f6526f95c66b74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:55:01.305121 11709 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86b680d6b9544f52b2f6526f95c66b74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:55:01.305289 11709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86b680d6b9544f52b2f6526f95c66b74 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:55:01.307708 11709 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86b680d6b9544f52b2f6526f95c66b74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b680d6b9544f52b2f6526f95c66b74" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40519 } }
I20260925 04:55:01.308271 11709 leader_election.cc:302] T 00000000000000000000000000000000 P 86b680d6b9544f52b2f6526f95c66b74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86b680d6b9544f52b2f6526f95c66b74; no voters:
I20260925 04:55:01.308876 11709 leader_election.cc:288] T 00000000000000000000000000000000 P 86b680d6b9544f52b2f6526f95c66b74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:55:01.309218 11714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86b680d6b9544f52b2f6526f95c66b74 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:55:01.309950 11714 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86b680d6b9544f52b2f6526f95c66b74 [term 1 LEADER]: Becoming Leader. State: Replica: 86b680d6b9544f52b2f6526f95c66b74, State: Running, Role: LEADER
I20260925 04:55:01.310437 11714 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86b680d6b9544f52b2f6526f95c66b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b680d6b9544f52b2f6526f95c66b74" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40519 } }
I20260925 04:55:01.311203 11709 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86b680d6b9544f52b2f6526f95c66b74 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:55:01.313397 11715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86b680d6b9544f52b2f6526f95c66b74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86b680d6b9544f52b2f6526f95c66b74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b680d6b9544f52b2f6526f95c66b74" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40519 } } }
I20260925 04:55:01.313817 11715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86b680d6b9544f52b2f6526f95c66b74 [sys.catalog]: This master's current role is: LEADER
I20260925 04:55:01.313500 11716 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86b680d6b9544f52b2f6526f95c66b74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86b680d6b9544f52b2f6526f95c66b74. Latest consensus state: current_term: 1 leader_uuid: "86b680d6b9544f52b2f6526f95c66b74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b680d6b9544f52b2f6526f95c66b74" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40519 } } }
I20260925 04:55:01.313957 11716 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86b680d6b9544f52b2f6526f95c66b74 [sys.catalog]: This master's current role is: LEADER
I20260925 04:55:01.315970 11719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:55:01.321736 11719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:55:01.331527 11719 catalog_manager.cc:1347] Generated new cluster ID: 081e2927f1464667b7f8cadf5affa2e1
I20260925 04:55:01.331712 11719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:55:01.348904 11719 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:55:01.350234 11719 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:55:01.360021 11719 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86b680d6b9544f52b2f6526f95c66b74: Generated new TSK 0
I20260925 04:55:01.360917 11719 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:55:01.382699 11712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:01.374855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32920 (local address 127.6.29.190:40519)
0925 04:55:01.375065 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:01.375074 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:01.375474 (+ 400us) server_negotiation.cc:407] Connection header received
0925 04:55:01.375754 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:01.375803 (+ 49us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:01.375909 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:01.376087 (+ 178us) server_negotiation.cc:226] Negotiated authn=JWT
0925 04:55:01.376571 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:01.377300 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:01.378724 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:01.379171 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:01.379327 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:01.379930 (+ 603us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 04:55:01.382048 (+ 2118us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34009/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 04:55:01.382384 (+ 336us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:32920: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34009/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 04:55:01.382876 11712 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:32920: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34009/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 04:55:01.383307 11738 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:01.374747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:40519 (local address 127.0.0.1:32920)
0925 04:55:01.375353 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:01.375370 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:01.375552 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:01.376097 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:01.376102 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:01.376114 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0925 04:55:01.376402 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:01.376411 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:01.377456 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:01.377463 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:01.378551 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:01.378563 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:01.379413 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:01.379427 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:01.379566 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:01.379601 (+ 35us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 04:55:01.382264 (+ 2663us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34009/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 04:55:01.382526 (+ 262us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.29.190:40519: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34009/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":325,"threads_started":1}
I20260925 04:55:01.384773 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 11643
I20260925 04:55:01.385926 11704 generic_service.cc:196] Checking for leaks (request via RPC)
E20260925 04:55:01.395453 11739 webserver.cc:533] Webserver: sslize failed:
E20260925 04:55:01.395597 11739 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 04:55:01.477629 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 11643
2026-09-25T04:55:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (714 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T04:55:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:55:01Z Disabled control of system clock
I20260925 04:55:01.561300 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:37181
--webserver_interface=127.6.29.190
--webserver_port=0
--builtin_ntp_servers=127.6.29.148:37837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:37181
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312053666503-6262-0/testchainca.cert with env {}
W20260925 04:55:01.888393 11744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:55:01.888720 11744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:55:01.888875 11744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:55:01.896550 11744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:55:01.896677 11744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:55:01.896713 11744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:55:01.896763 11744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:55:01.896800 11744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 04:55:01.896853 11744 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 04:55:01.896893 11744 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 04:55:01.907179 11744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:37837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:37181
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:37181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312053666503-6262-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:55:01.908558 11744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:55:01.909927 11744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:55:01.922384 11749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:01.922369 11750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:01.922497 11752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:55:01.923166 11744 server_base.cc:1034] running on GCE node
I20260925 04:55:01.924036 11744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:55:01.925529 11744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:55:01.926748 11744 hybrid_clock.cc:648] HybridClock initialized: now 1790312101926685 us; error 60 us; skew 500 ppm
I20260925 04:55:01.927135 11744 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 04:55:01.929531 11744 webserver.cc:460] Webserver started at http://127.6.29.190:42813/ using document root <none> and password file <none>
I20260925 04:55:01.930269 11744 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:01.930379 11744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:01.930709 11744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:55:01.933678 11744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "ece88432d822475c80f3053e6e8f7e71"
format_stamp: "Formatted at 2026-09-25 04:55:01 on dist-test-slave-xzjr"
server_key: "9e28dc0bcd80c00d12d69994a8efd98b"
server_key_iv: "9bacc9ec835863b23d48b93c09cd64e4"
server_key_version: "encryptionkey@0"
I20260925 04:55:01.934476 11744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "ece88432d822475c80f3053e6e8f7e71"
format_stamp: "Formatted at 2026-09-25 04:55:01 on dist-test-slave-xzjr"
server_key: "9e28dc0bcd80c00d12d69994a8efd98b"
server_key_iv: "9bacc9ec835863b23d48b93c09cd64e4"
server_key_version: "encryptionkey@0"
I20260925 04:55:01.939698 11744 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 04:55:01.943112 11758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:01.944585 11744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:55:01.944801 11744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "ece88432d822475c80f3053e6e8f7e71"
format_stamp: "Formatted at 2026-09-25 04:55:01 on dist-test-slave-xzjr"
server_key: "9e28dc0bcd80c00d12d69994a8efd98b"
server_key_iv: "9bacc9ec835863b23d48b93c09cd64e4"
server_key_version: "encryptionkey@0"
I20260925 04:55:01.944988 11744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:01.965278 11744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:55:01.966187 11744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:55:01.966478 11744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:55:01.985023 11744 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:37181
I20260925 04:55:01.985030 11809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:37181 every 8 connection(s)
I20260925 04:55:01.986411 11744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:55:01.990187 11810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:55:01.991138 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 11744
I20260925 04:55:01.991370 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 04:55:01.992074 6262 external_mini_cluster.cc:1428] Setting key b402f621e7aaea2738fcb3be82c5f3a1
I20260925 04:55:02.004113 11810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ece88432d822475c80f3053e6e8f7e71: Bootstrap starting.
I20260925 04:55:02.005860 11813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:01.994021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41084 (local address 127.6.29.190:37181)
0925 04:55:01.995103 (+ 1082us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:01.995114 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:01.995165 (+ 51us) server_negotiation.cc:407] Connection header received
0925 04:55:01.996776 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:01.996799 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:01.997295 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:01.997615 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:01.998205 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:01.999496 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:02.001401 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:02.002114 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:02.002264 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:02.003496 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:02.003504 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:02.003518 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:55:02.004125 (+ 607us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:02.004243 (+ 118us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:55:02.004273 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:02.004366 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:02.004585 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 04:55:02.004916 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":355,"threads_started":1}
I20260925 04:55:02.007364 11810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ece88432d822475c80f3053e6e8f7e71: Neither blocks nor log segments found. Creating new log.
I20260925 04:55:02.008495 11810 log.cc:824] T 00000000000000000000000000000000 P ece88432d822475c80f3053e6e8f7e71: Log is configured to *not* fsync() on all Append() calls
I20260925 04:55:02.009626 6262 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:55:02.011336 11810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ece88432d822475c80f3053e6e8f7e71: No bootstrap required, opened a new log
I20260925 04:55:02.018849 11810 raft_consensus.cc:348] T 00000000000000000000000000000000 P ece88432d822475c80f3053e6e8f7e71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece88432d822475c80f3053e6e8f7e71" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 37181 } }
I20260925 04:55:02.019250 11810 raft_consensus.cc:374] T 00000000000000000000000000000000 P ece88432d822475c80f3053e6e8f7e71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:55:02.019368 11810 raft_consensus.cc:729] T 00000000000000000000000000000000 P ece88432d822475c80f3053e6e8f7e71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ece88432d822475c80f3053e6e8f7e71, State: Initialized, Role: FOLLOWER
I20260925 04:55:02.019994 11810 consensus_queue.cc:260] T 00000000000000000000000000000000 P ece88432d822475c80f3053e6e8f7e71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece88432d822475c80f3053e6e8f7e71" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 37181 } }
I20260925 04:55:02.020224 11810 raft_consensus.cc:388] T 00000000000000000000000000000000 P ece88432d822475c80f3053e6e8f7e71 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:55:02.020409 11810 raft_consensus.cc:482] T 00000000000000000000000000000000 P ece88432d822475c80f3053e6e8f7e71 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:55:02.020581 11810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ece88432d822475c80f3053e6e8f7e71 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:55:02.022708 11810 raft_consensus.cc:504] T 00000000000000000000000000000000 P ece88432d822475c80f3053e6e8f7e71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece88432d822475c80f3053e6e8f7e71" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 37181 } }
I20260925 04:55:02.023243 11810 leader_election.cc:302] T 00000000000000000000000000000000 P ece88432d822475c80f3053e6e8f7e71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ece88432d822475c80f3053e6e8f7e71; no voters:
I20260925 04:55:02.023957 11810 leader_election.cc:288] T 00000000000000000000000000000000 P ece88432d822475c80f3053e6e8f7e71 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:55:02.024078 11815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ece88432d822475c80f3053e6e8f7e71 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:55:02.024819 11815 raft_consensus.cc:686] T 00000000000000000000000000000000 P ece88432d822475c80f3053e6e8f7e71 [term 1 LEADER]: Becoming Leader. State: Replica: ece88432d822475c80f3053e6e8f7e71, State: Running, Role: LEADER
I20260925 04:55:02.025327 11815 consensus_queue.cc:237] T 00000000000000000000000000000000 P ece88432d822475c80f3053e6e8f7e71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece88432d822475c80f3053e6e8f7e71" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 37181 } }
I20260925 04:55:02.026152 11810 sys_catalog.cc:564] T 00000000000000000000000000000000 P ece88432d822475c80f3053e6e8f7e71 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:55:02.028266 11816 sys_catalog.cc:455] T 00000000000000000000000000000000 P ece88432d822475c80f3053e6e8f7e71 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ece88432d822475c80f3053e6e8f7e71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece88432d822475c80f3053e6e8f7e71" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 37181 } } }
I20260925 04:55:02.028649 11816 sys_catalog.cc:458] T 00000000000000000000000000000000 P ece88432d822475c80f3053e6e8f7e71 [sys.catalog]: This master's current role is: LEADER
I20260925 04:55:02.030437 11817 sys_catalog.cc:455] T 00000000000000000000000000000000 P ece88432d822475c80f3053e6e8f7e71 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ece88432d822475c80f3053e6e8f7e71. Latest consensus state: current_term: 1 leader_uuid: "ece88432d822475c80f3053e6e8f7e71" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ece88432d822475c80f3053e6e8f7e71" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 37181 } } }
I20260925 04:55:02.030768 11817 sys_catalog.cc:458] T 00000000000000000000000000000000 P ece88432d822475c80f3053e6e8f7e71 [sys.catalog]: This master's current role is: LEADER
I20260925 04:55:02.031170 11824 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:55:02.036170 11824 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:55:02.045461 11824 catalog_manager.cc:1347] Generated new cluster ID: 6502cf6d56534ebb9f988c67d61d6d9c
I20260925 04:55:02.045600 11824 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:55:02.069309 11824 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:55:02.070722 11824 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:55:02.078239 11824 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ece88432d822475c80f3053e6e8f7e71: Generated new TSK 0
I20260925 04:55:02.079178 11824 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:55:02.189875 11813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:02.099607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41096 (local address 127.6.29.190:37181)
0925 04:55:02.099852 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:02.099860 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:02.100150 (+ 290us) server_negotiation.cc:407] Connection header received
0925 04:55:02.100408 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:02.100412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:02.100488 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:02.100648 (+ 160us) server_negotiation.cc:226] Negotiated authn=JWT
0925 04:55:02.101149 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:02.101876 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:02.103265 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:02.103731 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:02.103912 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:02.104392 (+ 480us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 04:55:02.189116 (+ 84724us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 04:55:02.189551 (+ 435us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41096: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 04:55:02.190057 11813 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41096: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 04:55:02.190166 11839 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:02.099441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:37181 (local address 127.0.0.1:41096)
0925 04:55:02.100035 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:02.100056 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:02.100229 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:02.100667 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:02.100672 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:02.100684 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0925 04:55:02.100983 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:02.100992 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:02.102026 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:02.102032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:02.103120 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:02.103131 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:02.103959 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:02.103965 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:02.104097 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:02.104128 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 04:55:02.189411 (+ 85283us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 04:55:02.189719 (+ 308us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.29.190:37181: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":433,"threads_started":1}
I20260925 04:55:02.191636 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 11744
I20260925 04:55:02.192854 11805 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:55:02.298614 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 11744
2026-09-25T04:55:02Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (802 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (418 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1481 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:55:04 dist-test-slave-xzjr krb5kdc[11855](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:55:04 dist-test-slave-xzjr krb5kdc[11855](info): set up 2 sockets
Sep 25 04:55:04 dist-test-slave-xzjr krb5kdc[11855](info): commencing operation
krb5kdc: starting...
W20260925 04:55:05.293147 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 04:55:05 dist-test-slave-xzjr krb5kdc[11855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:55:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:55:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.29.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
I20260925 04:55:05.507423 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:45691
--webserver_interface=127.6.29.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:34065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:45691
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:55:05.836370 11868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:55:05.836733 11868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:55:05.836901 11868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:55:05.844327 11868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:55:05.844482 11868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:55:05.844604 11868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:55:05.844702 11868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:55:05.844789 11868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:55:05.855382 11868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:34065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:45691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:45691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:55:05.856878 11868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:55:05.858300 11868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:55:05.870836 11873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:05.870702 11874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:05.870702 11876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:55:05.871503 11868 server_base.cc:1034] running on GCE node
I20260925 04:55:05.872457 11868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:55:05.873955 11868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:55:05.875226 11868 hybrid_clock.cc:648] HybridClock initialized: now 1790312105875169 us; error 51 us; skew 500 ppm
I20260925 04:55:05.875648 11868 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:55:05 dist-test-slave-xzjr krb5kdc[11855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312105, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:55:05.879015 11868 init.cc:377] Logged in from keytab as kudu/127.6.29.190@KRBTEST.COM (short username kudu)
I20260925 04:55:05.880766 11868 webserver.cc:460] Webserver started at http://127.6.29.190:39879/ using document root <none> and password file <none>
I20260925 04:55:05.881515 11868 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:05.881681 11868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:05.882067 11868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:55:05.885314 11868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "b1789e27604448c9b55f5f381f6f21d2"
format_stamp: "Formatted at 2026-09-25 04:55:05 on dist-test-slave-xzjr"
server_key: "d1780a41e51177edc2b454444e3c5629"
server_key_iv: "b303a478b305ec3d9c1b85e4f06ad43e"
server_key_version: "encryptionkey@0"
I20260925 04:55:05.886137 11868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "b1789e27604448c9b55f5f381f6f21d2"
format_stamp: "Formatted at 2026-09-25 04:55:05 on dist-test-slave-xzjr"
server_key: "d1780a41e51177edc2b454444e3c5629"
server_key_iv: "b303a478b305ec3d9c1b85e4f06ad43e"
server_key_version: "encryptionkey@0"
I20260925 04:55:05.891680 11868 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 04:55:05.895445 11883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:05.896999 11868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 04:55:05.897189 11868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "b1789e27604448c9b55f5f381f6f21d2"
format_stamp: "Formatted at 2026-09-25 04:55:05 on dist-test-slave-xzjr"
server_key: "d1780a41e51177edc2b454444e3c5629"
server_key_iv: "b303a478b305ec3d9c1b85e4f06ad43e"
server_key_version: "encryptionkey@0"
I20260925 04:55:05.897394 11868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:05.928923 11868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:55:05.933737 11868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:55:05.934051 11868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:55:05.953063 11868 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:45691
I20260925 04:55:05.953094 11934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:45691 every 8 connection(s)
I20260925 04:55:05.954613 11868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:55:05.958419 11935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:55:05.963940 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 11868
I20260925 04:55:05.964169 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 04:55:05.964767 6262 external_mini_cluster.cc:1428] Setting key fb52206bcf3b5dc7e89e7e6e64167c03
I20260925 04:55:05.969380 11935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1789e27604448c9b55f5f381f6f21d2: Bootstrap starting.
I20260925 04:55:05.973095 11935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1789e27604448c9b55f5f381f6f21d2: Neither blocks nor log segments found. Creating new log.
I20260925 04:55:05.974325 11935 log.cc:824] T 00000000000000000000000000000000 P b1789e27604448c9b55f5f381f6f21d2: Log is configured to *not* fsync() on all Append() calls
I20260925 04:55:05.977758 11935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1789e27604448c9b55f5f381f6f21d2: No bootstrap required, opened a new log
Sep 25 04:55:05 dist-test-slave-xzjr krb5kdc[11855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:55:05.988395 11935 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1789e27604448c9b55f5f381f6f21d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1789e27604448c9b55f5f381f6f21d2" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 45691 } }
I20260925 04:55:05.988891 11935 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1789e27604448c9b55f5f381f6f21d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:55:05.989003 11935 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1789e27604448c9b55f5f381f6f21d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1789e27604448c9b55f5f381f6f21d2, State: Initialized, Role: FOLLOWER
I20260925 04:55:05.989710 11935 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1789e27604448c9b55f5f381f6f21d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1789e27604448c9b55f5f381f6f21d2" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 45691 } }
I20260925 04:55:05.990005 11935 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1789e27604448c9b55f5f381f6f21d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:55:05.990141 11935 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1789e27604448c9b55f5f381f6f21d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:55:05.990317 11935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1789e27604448c9b55f5f381f6f21d2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:55:05.990417 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:05.966817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47636 (local address 127.6.29.190:45691)
0925 04:55:05.967631 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:05.967642 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:05.967683 (+ 41us) server_negotiation.cc:407] Connection header received
0925 04:55:05.969628 (+ 1945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:05.969651 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:05.970047 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:05.970393 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:05.972175 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:05.973510 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:05.975292 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:05.976224 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:05.976364 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:05.981138 (+ 4774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:05.981148 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:05.981184 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:05.981230 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:05.985523 (+ 4293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:05.986280 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:05.986290 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:05.986300 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:05.986541 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:05.986954 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:05.986959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:05.986962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:05.987802 (+ 840us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:05.988093 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:05.988448 (+ 355us) server_negotiation.cc:299] Negotiation successful
0925 04:55:05.988917 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":393,"threads_started":1}
I20260925 04:55:05.992892 11935 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1789e27604448c9b55f5f381f6f21d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1789e27604448c9b55f5f381f6f21d2" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 45691 } }
I20260925 04:55:05.993405 11935 leader_election.cc:302] T 00000000000000000000000000000000 P b1789e27604448c9b55f5f381f6f21d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1789e27604448c9b55f5f381f6f21d2; no voters:
I20260925 04:55:05.993992 11935 leader_election.cc:288] T 00000000000000000000000000000000 P b1789e27604448c9b55f5f381f6f21d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:55:05.994118 11940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1789e27604448c9b55f5f381f6f21d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:55:05.994866 11940 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1789e27604448c9b55f5f381f6f21d2 [term 1 LEADER]: Becoming Leader. State: Replica: b1789e27604448c9b55f5f381f6f21d2, State: Running, Role: LEADER
I20260925 04:55:05.995422 11940 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1789e27604448c9b55f5f381f6f21d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1789e27604448c9b55f5f381f6f21d2" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 45691 } }
I20260925 04:55:05.996179 11935 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1789e27604448c9b55f5f381f6f21d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:55:05.998610 11942 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1789e27604448c9b55f5f381f6f21d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1789e27604448c9b55f5f381f6f21d2. Latest consensus state: current_term: 1 leader_uuid: "b1789e27604448c9b55f5f381f6f21d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1789e27604448c9b55f5f381f6f21d2" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 45691 } } }
I20260925 04:55:05.999079 11942 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1789e27604448c9b55f5f381f6f21d2 [sys.catalog]: This master's current role is: LEADER
I20260925 04:55:05.999603 11941 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1789e27604448c9b55f5f381f6f21d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1789e27604448c9b55f5f381f6f21d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1789e27604448c9b55f5f381f6f21d2" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 45691 } } }
I20260925 04:55:06.000087 11941 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1789e27604448c9b55f5f381f6f21d2 [sys.catalog]: This master's current role is: LEADER
I20260925 04:55:06.004618 11948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:55:06.010042 11948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:55:06.020135 11948 catalog_manager.cc:1347] Generated new cluster ID: d166f8bc47fd459fb5420beea937a835
I20260925 04:55:06.020287 11948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:55:06.030709 11948 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:55:06.032163 11948 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:55:06.047057 11948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1789e27604448c9b55f5f381f6f21d2: Generated new TSK 0
I20260925 04:55:06.048055 11948 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.29.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab.
Entry for principal kudu/127.6.29.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab.
I20260925 04:55:06.108907 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.129:0
--local_ip_for_outbound_sockets=127.6.29.129
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:45691
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:34065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:55:06.451627 11961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:55:06.451990 11961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:55:06.452159 11961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:55:06.459435 11961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:55:06.459550 11961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:55:06.459736 11961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.129
I20260925 04:55:06.469791 11961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:34065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:45691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.129
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:55:06.471133 11961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:55:06.472483 11961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:55:06.486090 11967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:06.486204 11969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:06.486064 11966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:55:06.486872 11961 server_base.cc:1034] running on GCE node
I20260925 04:55:06.487449 11961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:55:06.488441 11961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:55:06.489682 11961 hybrid_clock.cc:648] HybridClock initialized: now 1790312106489592 us; error 51 us; skew 500 ppm
I20260925 04:55:06.490033 11961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:55:06 dist-test-slave-xzjr krb5kdc[11855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312106, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:55:06.493309 11961 init.cc:377] Logged in from keytab as kudu/127.6.29.129@KRBTEST.COM (short username kudu)
I20260925 04:55:06.494902 11961 webserver.cc:460] Webserver started at http://127.6.29.129:38141/ using document root <none> and password file <none>
I20260925 04:55:06.495616 11961 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:06.495828 11961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:06.496158 11961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:55:06.499344 11961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-0/data/instance:
uuid: "61ce6c1b828b4751943ff25ad8595329"
format_stamp: "Formatted at 2026-09-25 04:55:06 on dist-test-slave-xzjr"
server_key: "8b43dc46b693cd404868b5a8fda090b6"
server_key_iv: "631638496da52e9c4a211ffad3b681db"
server_key_version: "encryptionkey@0"
I20260925 04:55:06.500212 11961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance:
uuid: "61ce6c1b828b4751943ff25ad8595329"
format_stamp: "Formatted at 2026-09-25 04:55:06 on dist-test-slave-xzjr"
server_key: "8b43dc46b693cd404868b5a8fda090b6"
server_key_iv: "631638496da52e9c4a211ffad3b681db"
server_key_version: "encryptionkey@0"
I20260925 04:55:06.506052 11961 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260925 04:55:06.509788 11976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:06.511237 11961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:55:06.511425 11961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-0/wal
uuid: "61ce6c1b828b4751943ff25ad8595329"
format_stamp: "Formatted at 2026-09-25 04:55:06 on dist-test-slave-xzjr"
server_key: "8b43dc46b693cd404868b5a8fda090b6"
server_key_iv: "631638496da52e9c4a211ffad3b681db"
server_key_version: "encryptionkey@0"
I20260925 04:55:06.511646 11961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:06.531337 11961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:55:06.535502 11961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:55:06.535877 11961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:55:06.536865 11961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:55:06.538268 11961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:55:06.538371 11961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:06.538540 11961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:55:06.538616 11961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:06.572729 11961 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.129:35463
I20260925 04:55:06.572780 12088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.129:35463 every 8 connection(s)
I20260925 04:55:06.574127 11961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
I20260925 04:55:06.580669 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 11961
I20260925 04:55:06.580909 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance
I20260925 04:55:06.581540 6262 external_mini_cluster.cc:1428] Setting key a169f66c9cb9e76a62429f82d78aba9c
Sep 25 04:55:06 dist-test-slave-xzjr krb5kdc[11855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312106, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:55:06.595433 12091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:06.576903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:35745 (local address 127.6.29.190:45691)
0925 04:55:06.577397 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:06.577401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:06.578299 (+ 898us) server_negotiation.cc:407] Connection header received
0925 04:55:06.580165 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:06.580170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:06.580282 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:06.580401 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:06.582085 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:06.582972 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:06.584755 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:06.585515 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:06.585679 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:06.589383 (+ 3704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:06.589392 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:06.589399 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:06.589444 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:06.592093 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:06.593039 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:06.593043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:06.593045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:06.593187 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:06.593673 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:06.593677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:06.593679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:06.593938 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:06.594047 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:06.594982 (+ 935us) server_negotiation.cc:299] Negotiation successful
0925 04:55:06.595173 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":285,"threads_started":1}
I20260925 04:55:06.597051 12092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:06.577370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45691 (local address 127.6.29.129:35745)
0925 04:55:06.578142 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:06.578180 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:06.579893 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:06.580654 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:06.580663 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:06.581282 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:06.581915 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:06.581933 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:06.583184 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:06.583190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:06.584597 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:06.584609 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:06.585760 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:06.585765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:06.585971 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:06.586702 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:06.586730 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:06.589125 (+ 2395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:06.592234 (+ 3109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:06.592247 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:06.592268 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:06.592915 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:06.593290 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:06.593298 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:06.593304 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:06.593550 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:06.594066 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:06.594077 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:06.594558 (+ 481us) client_negotiation.cc:769] Sending connection context
0925 04:55:06.594872 (+ 314us) client_negotiation.cc:240] Negotiation successful
0925 04:55:06.595459 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":346,"threads_started":1}
I20260925 04:55:06.598757 12089 heartbeater.cc:347] Connected to a master server at 127.6.29.190:45691
I20260925 04:55:06.599084 12089 heartbeater.cc:464] Registering TS with master...
I20260925 04:55:06.599922 12089 heartbeater.cc:511] Master 127.6.29.190:45691 requested a full tablet report, sending...
I20260925 04:55:06.602249 11900 ts_manager.cc:194] Registered new tserver with Master: 61ce6c1b828b4751943ff25ad8595329 (127.6.29.129:35463)
WARNING: no policy specified for kudu/127.6.29.130@KRBTEST.COM; defaulting to no policy
I20260925 04:55:06.603905 11900 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.129@KRBTEST.COM'} at 127.6.29.129:35745
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab.
Entry for principal kudu/127.6.29.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab.
I20260925 04:55:06.631095 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.130:0
--local_ip_for_outbound_sockets=127.6.29.130
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:45691
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:34065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:55:06.964371 12095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:55:06.964751 12095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:55:06.964890 12095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:55:06.976500 12095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:55:06.976672 12095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:55:06.976821 12095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.130
I20260925 04:55:06.987385 12095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:34065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:45691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.130
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:55:06.988788 12095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:55:06.990051 12095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:55:07.003659 12100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:07.003708 12101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:07.004240 12103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:55:07.004679 12095 server_base.cc:1034] running on GCE node
I20260925 04:55:07.005323 12095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:55:07.006405 12095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:55:07.007671 12095 hybrid_clock.cc:648] HybridClock initialized: now 1790312107007635 us; error 34 us; skew 500 ppm
I20260925 04:55:07.008049 12095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:55:07 dist-test-slave-xzjr krb5kdc[11855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312107, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:55:07.015552 12095 init.cc:377] Logged in from keytab as kudu/127.6.29.130@KRBTEST.COM (short username kudu)
I20260925 04:55:07.017241 12095 webserver.cc:460] Webserver started at http://127.6.29.130:42201/ using document root <none> and password file <none>
I20260925 04:55:07.017971 12095 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:07.018116 12095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:07.018433 12095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:55:07.021457 12095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-1/data/instance:
uuid: "db425107592f4c23ab5e6cda2400db7a"
format_stamp: "Formatted at 2026-09-25 04:55:07 on dist-test-slave-xzjr"
server_key: "39e4cb60397afb68e695826b982485f4"
server_key_iv: "939416fb3d4ec782202cbc003c4dcc8a"
server_key_version: "encryptionkey@0"
I20260925 04:55:07.022271 12095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance:
uuid: "db425107592f4c23ab5e6cda2400db7a"
format_stamp: "Formatted at 2026-09-25 04:55:07 on dist-test-slave-xzjr"
server_key: "39e4cb60397afb68e695826b982485f4"
server_key_iv: "939416fb3d4ec782202cbc003c4dcc8a"
server_key_version: "encryptionkey@0"
I20260925 04:55:07.027653 12095 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 04:55:07.031374 12110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:07.032915 12095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 04:55:07.033094 12095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-1/wal
uuid: "db425107592f4c23ab5e6cda2400db7a"
format_stamp: "Formatted at 2026-09-25 04:55:07 on dist-test-slave-xzjr"
server_key: "39e4cb60397afb68e695826b982485f4"
server_key_iv: "939416fb3d4ec782202cbc003c4dcc8a"
server_key_version: "encryptionkey@0"
I20260925 04:55:07.033306 12095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:07.053934 12095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:55:07.058434 12095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:55:07.058754 12095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:55:07.059736 12095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:55:07.061103 12095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:55:07.061208 12095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:07.061369 12095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:55:07.061429 12095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:07.095357 12095 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.130:45659
I20260925 04:55:07.095405 12222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.130:45659 every 8 connection(s)
I20260925 04:55:07.096863 12095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
I20260925 04:55:07.102393 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 12095
I20260925 04:55:07.102596 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance
I20260925 04:55:07.103003 6262 external_mini_cluster.cc:1428] Setting key 13cee14a1350d142ccbfa841b20eafde
Sep 25 04:55:07 dist-test-slave-xzjr krb5kdc[11855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312107, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:55:07.119320 12225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:07.099680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:43089 (local address 127.6.29.190:45691)
0925 04:55:07.100079 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:07.100084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:07.101217 (+ 1133us) server_negotiation.cc:407] Connection header received
0925 04:55:07.103183 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:07.103189 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:07.103289 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:07.103416 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:07.105387 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:07.106111 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:07.107918 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:07.108659 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:07.108808 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:07.113230 (+ 4422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:07.113235 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:07.113238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:07.113270 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:07.116120 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:07.116979 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:07.116992 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:07.116994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:07.117122 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:07.117576 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:07.117588 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:07.117591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:07.117846 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:07.117999 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:07.118727 (+ 728us) server_negotiation.cc:299] Negotiation successful
0925 04:55:07.118941 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":205,"threads_started":1}
I20260925 04:55:07.120388 12226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:07.100275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45691 (local address 127.6.29.130:43089)
0925 04:55:07.100990 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:07.101034 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:07.102896 (+ 1862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:07.103666 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:07.103675 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:07.104450 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:07.105219 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:07.105241 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:07.106277 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:07.106281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:07.107740 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:07.107756 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:07.108897 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:07.108906 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:07.109214 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:07.110006 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:07.110043 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:07.112964 (+ 2921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:07.116279 (+ 3315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:07.116289 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:07.116308 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:07.116847 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:07.117251 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:07.117256 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:07.117260 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:07.117457 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:07.117975 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:07.117983 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:07.118347 (+ 364us) client_negotiation.cc:769] Sending connection context
0925 04:55:07.118639 (+ 292us) client_negotiation.cc:240] Negotiation successful
0925 04:55:07.119207 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":289,"threads_started":1}
I20260925 04:55:07.122032 12223 heartbeater.cc:347] Connected to a master server at 127.6.29.190:45691
I20260925 04:55:07.122367 12223 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.6.29.131@KRBTEST.COM; defaulting to no policy
I20260925 04:55:07.123170 12223 heartbeater.cc:511] Master 127.6.29.190:45691 requested a full tablet report, sending...
I20260925 04:55:07.124789 11900 ts_manager.cc:194] Registered new tserver with Master: db425107592f4c23ab5e6cda2400db7a (127.6.29.130:45659)
I20260925 04:55:07.125610 11900 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.130@KRBTEST.COM'} at 127.6.29.130:43089
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab.
Entry for principal kudu/127.6.29.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab.
I20260925 04:55:07.155335 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.131:0
--local_ip_for_outbound_sockets=127.6.29.131
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:45691
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:34065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:55:07.486804 12229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:55:07.487145 12229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:55:07.487313 12229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:55:07.494741 12229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:55:07.494926 12229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:55:07.495102 12229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.131
I20260925 04:55:07.505788 12229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:34065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:45691
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.131
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:55:07.507194 12229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:55:07.508721 12229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:55:07.522662 12234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:07.522590 12235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:07.522627 12237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:55:07.523502 12229 server_base.cc:1034] running on GCE node
I20260925 04:55:07.524164 12229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:55:07.525115 12229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:55:07.526350 12229 hybrid_clock.cc:648] HybridClock initialized: now 1790312107526298 us; error 42 us; skew 500 ppm
I20260925 04:55:07.526713 12229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:55:07 dist-test-slave-xzjr krb5kdc[11855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312107, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:55:07.530027 12229 init.cc:377] Logged in from keytab as kudu/127.6.29.131@KRBTEST.COM (short username kudu)
I20260925 04:55:07.531505 12229 webserver.cc:460] Webserver started at http://127.6.29.131:46391/ using document root <none> and password file <none>
I20260925 04:55:07.532271 12229 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:07.532383 12229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:07.532673 12229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:55:07.535604 12229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-2/data/instance:
uuid: "f0318aea53b4407aa7f0ad4ae9f38b25"
format_stamp: "Formatted at 2026-09-25 04:55:07 on dist-test-slave-xzjr"
server_key: "d9dceda772ccb4f3fef456f2f072b7fc"
server_key_iv: "c3618314e41635c701511f771e1c2ca7"
server_key_version: "encryptionkey@0"
I20260925 04:55:07.536615 12229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance:
uuid: "f0318aea53b4407aa7f0ad4ae9f38b25"
format_stamp: "Formatted at 2026-09-25 04:55:07 on dist-test-slave-xzjr"
server_key: "d9dceda772ccb4f3fef456f2f072b7fc"
server_key_iv: "c3618314e41635c701511f771e1c2ca7"
server_key_version: "encryptionkey@0"
I20260925 04:55:07.541919 12229 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 04:55:07.545524 12244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:07.546978 12229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 04:55:07.547194 12229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-2/wal
uuid: "f0318aea53b4407aa7f0ad4ae9f38b25"
format_stamp: "Formatted at 2026-09-25 04:55:07 on dist-test-slave-xzjr"
server_key: "d9dceda772ccb4f3fef456f2f072b7fc"
server_key_iv: "c3618314e41635c701511f771e1c2ca7"
server_key_version: "encryptionkey@0"
I20260925 04:55:07.547417 12229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:07.559437 12229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:55:07.563968 12229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:55:07.564273 12229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:55:07.565275 12229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:55:07.566573 12229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:55:07.566677 12229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:07.566819 12229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:55:07.566891 12229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:07.600463 12229 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.131:37743
I20260925 04:55:07.600481 12356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.131:37743 every 8 connection(s)
I20260925 04:55:07.601824 12229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
I20260925 04:55:07.605902 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 12229
I20260925 04:55:07.606098 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance
I20260925 04:55:07.606546 6262 external_mini_cluster.cc:1428] Setting key f3f6c78d58e69ed9d4de7cd8da589dd6
I20260925 04:55:07.607371 12089 heartbeater.cc:503] Master 127.6.29.190:45691 was elected leader, sending a full tablet report...
Sep 25 04:55:07 dist-test-slave-xzjr krb5kdc[11855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312107, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:55:07.621490 12225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:07.604473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:50157 (local address 127.6.29.190:45691)
0925 04:55:07.604641 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:07.604645 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:07.605815 (+ 1170us) server_negotiation.cc:407] Connection header received
0925 04:55:07.607924 (+ 2109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:07.607928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:07.607992 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:07.608080 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:07.609811 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:07.610681 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:07.612243 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:07.612640 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:07.612745 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:07.616081 (+ 3336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:07.616086 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:07.616090 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:07.616120 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:07.618500 (+ 2380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:07.619158 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:07.619161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:07.619164 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:07.619284 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:07.619697 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:07.619700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:07.619702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:07.619997 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:07.620114 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:07.620860 (+ 746us) server_negotiation.cc:299] Negotiation successful
0925 04:55:07.621109 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 04:55:07.622601 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:07.604921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45691 (local address 127.6.29.131:50157)
0925 04:55:07.605643 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:07.605708 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:07.607599 (+ 1891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:07.608291 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:07.608299 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:07.608951 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:07.609645 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:07.609663 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:07.610810 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:07.610813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:07.612106 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:07.612117 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:07.612819 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:07.612824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:07.613008 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:07.613618 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:07.613639 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:07.615842 (+ 2203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:07.618638 (+ 2796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:07.618644 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:07.618658 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:07.619055 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:07.619404 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:07.619407 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:07.619410 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:07.619595 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:07.620132 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:07.620138 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:07.620451 (+ 313us) client_negotiation.cc:769] Sending connection context
0925 04:55:07.620721 (+ 270us) client_negotiation.cc:240] Negotiation successful
0925 04:55:07.621284 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":268,"threads_started":1}
I20260925 04:55:07.623935 12357 heartbeater.cc:347] Connected to a master server at 127.6.29.190:45691
I20260925 04:55:07.624204 12357 heartbeater.cc:464] Registering TS with master...
I20260925 04:55:07.624898 12357 heartbeater.cc:511] Master 127.6.29.190:45691 requested a full tablet report, sending...
I20260925 04:55:07.626264 11900 ts_manager.cc:194] Registered new tserver with Master: f0318aea53b4407aa7f0ad4ae9f38b25 (127.6.29.131:37743)
I20260925 04:55:07.627142 11900 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.131@KRBTEST.COM'} at 127.6.29.131:50157
I20260925 04:55:07.634658 6262 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/krb5kdc/test-admin.keytab.
Sep 25 04:55:07 dist-test-slave-xzjr krb5kdc[11855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:55:07.663686 6262 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 04:55:07.669225 12364 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:07.666400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56514 (local address 127.6.29.129:35463)
0925 04:55:07.667088 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:07.667098 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:07.667132 (+ 34us) server_negotiation.cc:407] Connection header received
0925 04:55:07.667296 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:07.667303 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:07.667551 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:07.667771 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:07.668777 (+ 1006us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56514: BlockingRecv error: recv got EOF from 127.0.0.1:56514 (error 108)
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":342,"threads_started":1}
W20260925 04:55:07.669354 6262 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/krb5kdc/krb5cc' of type FILE
Sep 25 04:55:07 dist-test-slave-xzjr krb5kdc[11855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:55:07.670972 6262 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 04:55:07 dist-test-slave-xzjr krb5kdc[11855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:55:07.672372 6262 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 04:55:07 dist-test-slave-xzjr krb5kdc[11855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.129@KRBTEST.COM
I20260925 04:55:07.688031 12364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:07.674105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56530 (local address 127.6.29.129:35463)
0925 04:55:07.674312 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:07.674316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:07.674448 (+ 132us) server_negotiation.cc:407] Connection header received
0925 04:55:07.674597 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:07.674600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:07.674657 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:07.674751 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:07.675693 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:07.676707 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:07.677928 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:07.678479 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:07.678585 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:07.682268 (+ 3683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:07.682283 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:07.682302 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:07.682340 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:07.685197 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:07.685852 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:07.685859 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:07.685862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:07.685977 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:07.686376 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:07.686379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:07.686381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:07.686766 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:07.686982 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:07.687338 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 04:55:07.687658 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260925 04:55:07.694588 12364 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:07.692560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56536 (local address 127.6.29.129:35463)
0925 04:55:07.692745 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:07.692749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:07.693089 (+ 340us) server_negotiation.cc:407] Connection header received
0925 04:55:07.693235 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:07.693239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:07.693306 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:07.693403 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:07.694356 (+ 953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56536: BlockingRecv error: recv got EOF from 127.0.0.1:56536 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260925 04:55:07.694847 6262 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/krb5kdc/krb5cc' of type FILE
Sep 25 04:55:07 dist-test-slave-xzjr krb5kdc[11855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:55:07.696456 6262 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 04:55:07 dist-test-slave-xzjr krb5kdc[11855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:55:07.697872 6262 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 04:55:07 dist-test-slave-xzjr krb5kdc[11855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.129@KRBTEST.COM
I20260925 04:55:07.713117 12364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:07.699225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56552 (local address 127.6.29.129:35463)
0925 04:55:07.699437 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:07.699443 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:07.699738 (+ 295us) server_negotiation.cc:407] Connection header received
0925 04:55:07.700045 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:07.700048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:07.700106 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:07.700197 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:07.701224 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:07.702017 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:07.703186 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:07.703583 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:07.703717 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:07.707530 (+ 3813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:07.707535 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:07.707538 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:07.707579 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:07.710385 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:07.711055 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:07.711059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:07.711061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:07.711188 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:07.711579 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:07.711583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:07.711585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:07.711977 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:07.712203 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:07.712593 (+ 390us) server_negotiation.cc:299] Negotiation successful
0925 04:55:07.712840 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260925 04:55:07.717733 12364 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:07.716002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56560 (local address 127.6.29.129:35463)
0925 04:55:07.716338 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:07.716344 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:07.716455 (+ 111us) server_negotiation.cc:407] Connection header received
0925 04:55:07.716641 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:07.716649 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:07.716745 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:07.716852 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:07.717501 (+ 649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56560: BlockingRecv error: recv got EOF from 127.0.0.1:56560 (error 108)
Metrics: {"server-negotiator.queue_time_us":229}
W20260925 04:55:07.718313 6262 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790312053666503-6262-0/krb5kdc/krb5cc)
Sep 25 04:55:07 dist-test-slave-xzjr krb5kdc[11855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:55:07.720147 6262 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 04:55:07 dist-test-slave-xzjr krb5kdc[11855](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:55:07.721537 6262 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 04:55:07 dist-test-slave-xzjr krb5kdc[11855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.129@KRBTEST.COM
I20260925 04:55:07.735459 12364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:07.722941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56574 (local address 127.6.29.129:35463)
0925 04:55:07.723143 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:07.723147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:07.723237 (+ 90us) server_negotiation.cc:407] Connection header received
0925 04:55:07.723342 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:07.723346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:07.723398 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:07.723477 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:07.724440 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:07.725127 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:07.726259 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:07.726672 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:07.726768 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:07.730372 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:07.730376 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:07.730380 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:07.730411 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:07.732995 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:07.733592 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:07.733595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:07.733598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:07.733739 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:07.734130 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:07.734133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:07.734135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:07.734402 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:07.734638 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:07.734911 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 04:55:07.735072 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 04:55:07.737298 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 11961
I20260925 04:55:07.749380 12364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:07.738087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56582 (local address 127.6.29.129:35463)
0925 04:55:07.738286 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:07.738295 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:07.738639 (+ 344us) server_negotiation.cc:407] Connection header received
0925 04:55:07.738803 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:07.738809 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:07.738873 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:07.738973 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:07.739961 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:07.740637 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:07.741825 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:07.742234 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:07.742361 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:07.744457 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:07.744463 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:07.744474 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:07.744503 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:07.746900 (+ 2397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:07.747519 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:07.747525 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:07.747528 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:07.747670 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:07.748071 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:07.748078 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:07.748081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:07.748326 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:07.748604 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:07.748875 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 04:55:07.749074 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 04:55:07.750061 12084 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:55:07.834379 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 11961
I20260925 04:55:07.850930 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 12095
Sep 25 04:55:07 dist-test-slave-xzjr krb5kdc[11855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.130@KRBTEST.COM
I20260925 04:55:07.866950 12377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:07.851963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54850 (local address 127.6.29.130:45659)
0925 04:55:07.852524 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:07.852534 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:07.852567 (+ 33us) server_negotiation.cc:407] Connection header received
0925 04:55:07.852639 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:07.852645 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:07.852869 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:07.853016 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:07.854114 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:07.855121 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:07.856516 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:07.857104 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:07.857215 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:07.861174 (+ 3959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:07.861188 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:07.861203 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:07.861243 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:07.864339 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:07.865048 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:07.865055 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:07.865058 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:07.865180 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:07.865521 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:07.865524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:07.865525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:07.865861 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:07.866082 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:07.866345 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 04:55:07.866609 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":304,"threads_started":1}
I20260925 04:55:07.868346 12218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:55:07.951570 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 12095
I20260925 04:55:07.976541 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 12229
Sep 25 04:55:07 dist-test-slave-xzjr krb5kdc[11855](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.131@KRBTEST.COM
I20260925 04:55:07.996794 12379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:07.977630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35054 (local address 127.6.29.131:37743)
0925 04:55:07.978334 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:07.978349 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:07.978379 (+ 30us) server_negotiation.cc:407] Connection header received
0925 04:55:07.978461 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:07.978471 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:07.978784 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:07.978973 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:07.980381 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:07.981465 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:07.982787 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:07.983745 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:07.984051 (+ 306us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:07.989139 (+ 5088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:07.989156 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:07.989176 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:07.989234 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:07.993282 (+ 4048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:07.994147 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:07.994159 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:07.994162 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:07.994326 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:07.994713 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:07.994722 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:07.994729 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:07.995278 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:07.995653 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:07.995920 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 04:55:07.996253 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":337,"threads_started":1}
I20260925 04:55:07.998492 12352 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:55:08.080556 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 12229
I20260925 04:55:08.095966 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 11868
I20260925 04:55:08.097219 11930 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:55:08.202997 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 11868
2026-09-25T04:55:08Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4009 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:55:08 dist-test-slave-xzjr krb5kdc[12384](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:55:08 dist-test-slave-xzjr krb5kdc[12384](info): set up 2 sockets
Sep 25 04:55:08 dist-test-slave-xzjr krb5kdc[12384](info): commencing operation
krb5kdc: starting...
W20260925 04:55:09.297646 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 04:55:09 dist-test-slave-xzjr krb5kdc[12384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:55:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:55:09Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.29.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.29.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.29.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.190.keytab.
Entry for principal oryx/127.6.29.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.190.keytab.
I20260925 04:55:09.511658 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:42115
--webserver_interface=127.6.29.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.190.keytab
--principal=oryx/127.6.29.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:44961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:42115
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:55:09.839107 12397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:55:09.839488 12397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:55:09.839668 12397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:55:09.847821 12397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:55:09.847990 12397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:55:09.848075 12397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:55:09.848167 12397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:55:09.848237 12397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:55:09.858232 12397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:44961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:42115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.190.keytab
--principal=oryx/127.6.29.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:42115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:55:09.859709 12397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:55:09.861146 12397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:55:09.873561 12405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:09.873658 12403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:09.873597 12402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:55:09.874650 12397 server_base.cc:1034] running on GCE node
I20260925 04:55:09.875483 12397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:55:09.877090 12397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:55:09.878365 12397 hybrid_clock.cc:648] HybridClock initialized: now 1790312109878303 us; error 66 us; skew 500 ppm
I20260925 04:55:09.878790 12397 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:55:09 dist-test-slave-xzjr krb5kdc[12384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312109, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.29.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:55:09.882068 12397 init.cc:377] Logged in from keytab as oryx/127.6.29.190@KRBTEST.COM (short username oryx)
I20260925 04:55:09.883687 12397 webserver.cc:460] Webserver started at http://127.6.29.190:35337/ using document root <none> and password file <none>
I20260925 04:55:09.884421 12397 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:09.884584 12397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:09.884904 12397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:55:09.888069 12397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "932891fca8784f3cbae340352149f1fb"
format_stamp: "Formatted at 2026-09-25 04:55:09 on dist-test-slave-xzjr"
server_key: "a8cedb76f1b816cc18ef27bcf4afa1ce"
server_key_iv: "8521e878c7825bc2de57aa14d6d8db8c"
server_key_version: "encryptionkey@0"
I20260925 04:55:09.888870 12397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "932891fca8784f3cbae340352149f1fb"
format_stamp: "Formatted at 2026-09-25 04:55:09 on dist-test-slave-xzjr"
server_key: "a8cedb76f1b816cc18ef27bcf4afa1ce"
server_key_iv: "8521e878c7825bc2de57aa14d6d8db8c"
server_key_version: "encryptionkey@0"
I20260925 04:55:09.894240 12397 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260925 04:55:09.897787 12412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:09.899222 12397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:55:09.899408 12397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "932891fca8784f3cbae340352149f1fb"
format_stamp: "Formatted at 2026-09-25 04:55:09 on dist-test-slave-xzjr"
server_key: "a8cedb76f1b816cc18ef27bcf4afa1ce"
server_key_iv: "8521e878c7825bc2de57aa14d6d8db8c"
server_key_version: "encryptionkey@0"
I20260925 04:55:09.899611 12397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:09.943064 12397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:55:09.947664 12397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:55:09.948001 12397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:55:09.966717 12397 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:42115
I20260925 04:55:09.966758 12463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:42115 every 8 connection(s)
I20260925 04:55:09.968159 12397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:55:09.971850 12464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:55:09.973155 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 12397
I20260925 04:55:09.973387 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 04:55:09.974184 6262 external_mini_cluster.cc:1428] Setting key 82e4f15cdb923ce632c50d96de858be4
I20260925 04:55:09.986642 12464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb: Bootstrap starting.
Sep 25 04:55:09 dist-test-slave-xzjr krb5kdc[12384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM
I20260925 04:55:09.990345 12464 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb: Neither blocks nor log segments found. Creating new log.
I20260925 04:55:09.991420 12464 log.cc:824] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb: Log is configured to *not* fsync() on all Append() calls
I20260925 04:55:09.994385 12464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb: No bootstrap required, opened a new log
I20260925 04:55:09.997884 12467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:09.976396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46514 (local address 127.6.29.190:42115)
0925 04:55:09.977284 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:09.977298 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:09.977346 (+ 48us) server_negotiation.cc:407] Connection header received
0925 04:55:09.978902 (+ 1556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:09.978925 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:09.979353 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:09.979692 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:09.981517 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:09.982825 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:09.984646 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:09.985337 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:09.985477 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:09.990269 (+ 4792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:09.990279 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:09.990294 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:09.990334 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:09.993632 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:09.994363 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:09.994370 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:09.994381 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:09.994637 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:09.995040 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:09.995044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:09.995050 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:09.995585 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:09.995954 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:09.996339 (+ 385us) server_negotiation.cc:299] Negotiation successful
0925 04:55:09.996825 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":387,"threads_started":1}
I20260925 04:55:10.002097 12464 raft_consensus.cc:348] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } }
I20260925 04:55:10.002482 12464 raft_consensus.cc:374] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:55:10.002633 12464 raft_consensus.cc:729] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 932891fca8784f3cbae340352149f1fb, State: Initialized, Role: FOLLOWER
I20260925 04:55:10.003196 12464 consensus_queue.cc:260] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } }
I20260925 04:55:10.003445 12464 raft_consensus.cc:388] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:55:10.003558 12464 raft_consensus.cc:482] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:55:10.003685 12464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:55:10.005999 12464 raft_consensus.cc:504] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } }
I20260925 04:55:10.006450 12464 leader_election.cc:302] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 932891fca8784f3cbae340352149f1fb; no voters:
I20260925 04:55:10.006981 12464 leader_election.cc:288] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:55:10.007160 12469 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:55:10.007956 12469 raft_consensus.cc:686] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [term 1 LEADER]: Becoming Leader. State: Replica: 932891fca8784f3cbae340352149f1fb, State: Running, Role: LEADER
I20260925 04:55:10.008704 12469 consensus_queue.cc:237] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } }
I20260925 04:55:10.009183 12464 sys_catalog.cc:564] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:55:10.011636 12470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "932891fca8784f3cbae340352149f1fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } }
I20260925 04:55:10.011619 12471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 932891fca8784f3cbae340352149f1fb. Latest consensus state: current_term: 1 leader_uuid: "932891fca8784f3cbae340352149f1fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } }
I20260925 04:55:10.012089 12470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [sys.catalog]: This master's current role is: LEADER
I20260925 04:55:10.012089 12471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [sys.catalog]: This master's current role is: LEADER
I20260925 04:55:10.012921 12474 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:55:10.018704 12474 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:55:10.027683 12474 catalog_manager.cc:1347] Generated new cluster ID: f48087faeea44abca781b012f0655bbc
I20260925 04:55:10.027832 12474 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:55:10.062142 12474 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:55:10.063558 12474 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:55:10.072454 12474 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb: Generated new TSK 0
I20260925 04:55:10.073330 12474 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.29.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.29.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.29.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.129.keytab.
Entry for principal oryx/127.6.29.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.129.keytab.
I20260925 04:55:10.136142 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.129:0
--local_ip_for_outbound_sockets=127.6.29.129
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:42115
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.129.keytab
--principal=oryx/127.6.29.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:44961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:55:10.479199 12490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:55:10.479566 12490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:55:10.479743 12490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:55:10.487190 12490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:55:10.487371 12490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:55:10.487573 12490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.129
I20260925 04:55:10.497768 12490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:44961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.129.keytab
--principal=oryx/127.6.29.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:42115
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.129
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:55:10.499179 12490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:55:10.500636 12490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:55:10.515188 12496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:10.515369 12495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:10.515941 12498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:55:10.516269 12490 server_base.cc:1034] running on GCE node
I20260925 04:55:10.517057 12490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:55:10.518285 12490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:55:10.519503 12490 hybrid_clock.cc:648] HybridClock initialized: now 1790312110519449 us; error 40 us; skew 500 ppm
I20260925 04:55:10.519941 12490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:55:10 dist-test-slave-xzjr krb5kdc[12384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312110, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.29.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:55:10.523631 12490 init.cc:377] Logged in from keytab as oryx/127.6.29.129@KRBTEST.COM (short username oryx)
I20260925 04:55:10.525271 12490 webserver.cc:460] Webserver started at http://127.6.29.129:33697/ using document root <none> and password file <none>
I20260925 04:55:10.526082 12490 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:10.526233 12490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:10.526578 12490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:55:10.530097 12490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-0/data/instance:
uuid: "dad67e0d8c3a4ab8adeda07f8db9a9a6"
format_stamp: "Formatted at 2026-09-25 04:55:10 on dist-test-slave-xzjr"
server_key: "8533d79dea6d8f192b28acb2e9f0afdb"
server_key_iv: "60dfc4667064e7cec8cddc1952b46bcf"
server_key_version: "encryptionkey@0"
I20260925 04:55:10.530956 12490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance:
uuid: "dad67e0d8c3a4ab8adeda07f8db9a9a6"
format_stamp: "Formatted at 2026-09-25 04:55:10 on dist-test-slave-xzjr"
server_key: "8533d79dea6d8f192b28acb2e9f0afdb"
server_key_iv: "60dfc4667064e7cec8cddc1952b46bcf"
server_key_version: "encryptionkey@0"
I20260925 04:55:10.537048 12490 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260925 04:55:10.541054 12505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:10.542647 12490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 04:55:10.542850 12490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-0/wal
uuid: "dad67e0d8c3a4ab8adeda07f8db9a9a6"
format_stamp: "Formatted at 2026-09-25 04:55:10 on dist-test-slave-xzjr"
server_key: "8533d79dea6d8f192b28acb2e9f0afdb"
server_key_iv: "60dfc4667064e7cec8cddc1952b46bcf"
server_key_version: "encryptionkey@0"
I20260925 04:55:10.543071 12490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:10.559615 12490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:55:10.564349 12490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:55:10.564662 12490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:55:10.565670 12490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:55:10.567124 12490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:55:10.567227 12490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:10.567384 12490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:55:10.567453 12490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:10.601210 12490 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.129:46555
I20260925 04:55:10.601269 12617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.129:46555 every 8 connection(s)
I20260925 04:55:10.602561 12490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
I20260925 04:55:10.610714 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 12490
I20260925 04:55:10.610944 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance
I20260925 04:55:10.611557 6262 external_mini_cluster.cc:1428] Setting key af19fdb7c047a53301028698c3da85f1
Sep 25 04:55:10 dist-test-slave-xzjr krb5kdc[12384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312110, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.29.129@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM
I20260925 04:55:10.623222 12620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:10.605263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:51713 (local address 127.6.29.190:42115)
0925 04:55:10.605711 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:10.605715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:10.606645 (+ 930us) server_negotiation.cc:407] Connection header received
0925 04:55:10.608565 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:10.608570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:10.608668 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:10.608801 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:10.610442 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:10.611325 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:10.613026 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:10.613439 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:10.613569 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:10.617337 (+ 3768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:10.617343 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:10.617346 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:10.617380 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:10.620109 (+ 2729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:10.620875 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:10.620880 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:10.620882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:10.620997 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:10.621441 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:10.621446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:10.621449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:10.621737 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:10.621906 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:10.622646 (+ 740us) server_negotiation.cc:299] Negotiation successful
0925 04:55:10.622868 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":235,"threads_started":1}
I20260925 04:55:10.624322 12621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:10.605756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.6.29.129:51713)
0925 04:55:10.606462 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:10.606503 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:10.608270 (+ 1767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:10.609076 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:10.609092 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:10.609619 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:10.610272 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:10.610290 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:10.611497 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:10.611501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:10.612869 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:10.612881 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:10.613671 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:10.613678 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:10.613927 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:10.614650 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:10.614674 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:10.617100 (+ 2426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:10.620257 (+ 3157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:10.620267 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:10.620286 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:10.620734 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:10.621115 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:10.621120 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:10.621123 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:10.621319 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:10.621868 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:10.621876 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:10.622255 (+ 379us) client_negotiation.cc:769] Sending connection context
0925 04:55:10.622534 (+ 279us) client_negotiation.cc:240] Negotiation successful
0925 04:55:10.623096 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":282,"threads_started":1}
I20260925 04:55:10.626048 12618 heartbeater.cc:347] Connected to a master server at 127.6.29.190:42115
I20260925 04:55:10.626390 12618 heartbeater.cc:464] Registering TS with master...
I20260925 04:55:10.627297 12618 heartbeater.cc:511] Master 127.6.29.190:42115 requested a full tablet report, sending...
I20260925 04:55:10.630180 12429 ts_manager.cc:194] Registered new tserver with Master: dad67e0d8c3a4ab8adeda07f8db9a9a6 (127.6.29.129:46555)
WARNING: no policy specified for oryx/127.6.29.130@KRBTEST.COM; defaulting to no policy
I20260925 04:55:10.632371 12429 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.29.129@KRBTEST.COM'} at 127.6.29.129:51713
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.29.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.29.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.130.keytab.
Entry for principal oryx/127.6.29.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.130.keytab.
I20260925 04:55:10.661367 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.130:0
--local_ip_for_outbound_sockets=127.6.29.130
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:42115
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.130.keytab
--principal=oryx/127.6.29.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:44961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:55:10.994784 12624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:55:10.995149 12624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:55:10.995311 12624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:55:11.002789 12624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:55:11.002978 12624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:55:11.003201 12624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.130
I20260925 04:55:11.013585 12624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:44961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.130.keytab
--principal=oryx/127.6.29.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:42115
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.130
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:55:11.014953 12624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:55:11.016431 12624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:55:11.030253 12629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:11.030335 12632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:11.030254 12630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:55:11.031122 12624 server_base.cc:1034] running on GCE node
I20260925 04:55:11.031812 12624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:55:11.032861 12624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:55:11.034092 12624 hybrid_clock.cc:648] HybridClock initialized: now 1790312111034044 us; error 44 us; skew 500 ppm
I20260925 04:55:11.034444 12624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:55:11 dist-test-slave-xzjr krb5kdc[12384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312111, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.29.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:55:11.037766 12624 init.cc:377] Logged in from keytab as oryx/127.6.29.130@KRBTEST.COM (short username oryx)
I20260925 04:55:11.039211 12624 webserver.cc:460] Webserver started at http://127.6.29.130:40581/ using document root <none> and password file <none>
I20260925 04:55:11.039963 12624 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:11.040066 12624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:11.040431 12624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:55:11.043437 12624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-1/data/instance:
uuid: "2f7cb902838c4ca382524ae7baa660e4"
format_stamp: "Formatted at 2026-09-25 04:55:11 on dist-test-slave-xzjr"
server_key: "a2eafab9293d8d5722689e6fdec0896a"
server_key_iv: "e190b75485b0b470fc6a7f3e677af2ee"
server_key_version: "encryptionkey@0"
I20260925 04:55:11.044232 12624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance:
uuid: "2f7cb902838c4ca382524ae7baa660e4"
format_stamp: "Formatted at 2026-09-25 04:55:11 on dist-test-slave-xzjr"
server_key: "a2eafab9293d8d5722689e6fdec0896a"
server_key_iv: "e190b75485b0b470fc6a7f3e677af2ee"
server_key_version: "encryptionkey@0"
I20260925 04:55:11.049793 12624 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 04:55:11.053391 12639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:11.054834 12624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 04:55:11.054999 12624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-1/wal
uuid: "2f7cb902838c4ca382524ae7baa660e4"
format_stamp: "Formatted at 2026-09-25 04:55:11 on dist-test-slave-xzjr"
server_key: "a2eafab9293d8d5722689e6fdec0896a"
server_key_iv: "e190b75485b0b470fc6a7f3e677af2ee"
server_key_version: "encryptionkey@0"
I20260925 04:55:11.055218 12624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:11.076037 12624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:55:11.080327 12624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:55:11.080624 12624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:55:11.081697 12624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:55:11.083105 12624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:55:11.083192 12624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:11.083324 12624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:55:11.083374 12624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:11.117944 12624 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.130:45893
I20260925 04:55:11.117990 12751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.130:45893 every 8 connection(s)
I20260925 04:55:11.119329 12624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
I20260925 04:55:11.122714 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 12624
I20260925 04:55:11.122922 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance
I20260925 04:55:11.123526 6262 external_mini_cluster.cc:1428] Setting key 88c0d0930317a77d0842b445f4eaa340
Sep 25 04:55:11 dist-test-slave-xzjr krb5kdc[12384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312111, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.29.130@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM
I20260925 04:55:11.142355 12620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:11.122150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:33749 (local address 127.6.29.190:42115)
0925 04:55:11.122354 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:11.122359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:11.123904 (+ 1545us) server_negotiation.cc:407] Connection header received
0925 04:55:11.126387 (+ 2483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:11.126392 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:11.126475 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:11.126600 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:11.128286 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:11.128987 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:11.130578 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:11.131000 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:11.131146 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:11.135681 (+ 4535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:11.135686 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:11.135690 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:11.135725 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:11.138527 (+ 2802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:11.139568 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:11.139572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:11.139575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:11.139742 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:11.140321 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:11.140325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:11.140327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:11.140718 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:11.140847 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:11.141908 (+ 1061us) server_negotiation.cc:299] Negotiation successful
0925 04:55:11.142098 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 04:55:11.144156 12754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:11.122727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.6.29.130:33749)
0925 04:55:11.123689 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:11.123730 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:11.126100 (+ 2370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:11.126865 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:11.126874 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:11.127388 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:11.128107 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:11.128127 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:11.129146 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:11.129152 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:11.130411 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:11.130423 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:11.131243 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:11.131250 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:11.131490 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:11.132273 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:11.132315 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:11.135410 (+ 3095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:11.138697 (+ 3287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:11.138710 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:11.138733 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:11.139432 (+ 699us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:11.139922 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:11.139930 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:11.139936 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:11.140205 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:11.140865 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:11.140875 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:11.141434 (+ 559us) client_negotiation.cc:769] Sending connection context
0925 04:55:11.141810 (+ 376us) client_negotiation.cc:240] Negotiation successful
0925 04:55:11.142512 (+ 702us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":415,"threads_started":1}
WARNING: no policy specified for oryx/127.6.29.131@KRBTEST.COM; defaulting to no policy
I20260925 04:55:11.146072 12752 heartbeater.cc:347] Connected to a master server at 127.6.29.190:42115
I20260925 04:55:11.146481 12752 heartbeater.cc:464] Registering TS with master...
I20260925 04:55:11.147523 12752 heartbeater.cc:511] Master 127.6.29.190:42115 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.29.131@KRBTEST.COM" created.
I20260925 04:55:11.149381 12429 ts_manager.cc:194] Registered new tserver with Master: 2f7cb902838c4ca382524ae7baa660e4 (127.6.29.130:45893)
I20260925 04:55:11.150233 12429 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.29.130@KRBTEST.COM'} at 127.6.29.130:33749
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.29.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.131.keytab.
Entry for principal oryx/127.6.29.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.131.keytab.
I20260925 04:55:11.176023 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.131:0
--local_ip_for_outbound_sockets=127.6.29.131
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:42115
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.131.keytab
--principal=oryx/127.6.29.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:44961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:55:11.503999 12757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:55:11.504351 12757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:55:11.504527 12757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:55:11.512130 12757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:55:11.512295 12757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:55:11.512485 12757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.131
I20260925 04:55:11.522652 12757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:44961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.131.keytab
--principal=oryx/127.6.29.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:42115
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.131
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:55:11.524284 12757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:55:11.525660 12757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:55:11.539206 12762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:11.539219 12763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:11.539268 12765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:55:11.545614 12757 server_base.cc:1034] running on GCE node
I20260925 04:55:11.546387 12757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:55:11.547554 12757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:55:11.548781 12757 hybrid_clock.cc:648] HybridClock initialized: now 1790312111548707 us; error 70 us; skew 500 ppm
I20260925 04:55:11.549153 12757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:55:11 dist-test-slave-xzjr krb5kdc[12384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312111, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.29.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:55:11.552433 12757 init.cc:377] Logged in from keytab as oryx/127.6.29.131@KRBTEST.COM (short username oryx)
I20260925 04:55:11.554141 12757 webserver.cc:460] Webserver started at http://127.6.29.131:37041/ using document root <none> and password file <none>
I20260925 04:55:11.554900 12757 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:11.555001 12757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:11.555272 12757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:55:11.558362 12757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-2/data/instance:
uuid: "fbde0dd6885c4cd9ab573d65bfc3231e"
format_stamp: "Formatted at 2026-09-25 04:55:11 on dist-test-slave-xzjr"
server_key: "9d1c0be5fde281a763df21cb7f672c18"
server_key_iv: "c2261f32244882c7bd81c455ed04d834"
server_key_version: "encryptionkey@0"
I20260925 04:55:11.559232 12757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance:
uuid: "fbde0dd6885c4cd9ab573d65bfc3231e"
format_stamp: "Formatted at 2026-09-25 04:55:11 on dist-test-slave-xzjr"
server_key: "9d1c0be5fde281a763df21cb7f672c18"
server_key_iv: "c2261f32244882c7bd81c455ed04d834"
server_key_version: "encryptionkey@0"
I20260925 04:55:11.564803 12757 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260925 04:55:11.568429 12772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:11.569959 12757 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:55:11.570132 12757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-2/wal
uuid: "fbde0dd6885c4cd9ab573d65bfc3231e"
format_stamp: "Formatted at 2026-09-25 04:55:11 on dist-test-slave-xzjr"
server_key: "9d1c0be5fde281a763df21cb7f672c18"
server_key_iv: "c2261f32244882c7bd81c455ed04d834"
server_key_version: "encryptionkey@0"
I20260925 04:55:11.570309 12757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:11.587297 12757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:55:11.591444 12757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:55:11.591711 12757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:55:11.592669 12757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:55:11.594041 12757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:55:11.594125 12757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:11.594211 12757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:55:11.594255 12757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:11.631399 12757 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.131:44161
I20260925 04:55:11.631449 12884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.131:44161 every 8 connection(s)
I20260925 04:55:11.632844 12757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
I20260925 04:55:11.635891 12618 heartbeater.cc:503] Master 127.6.29.190:42115 was elected leader, sending a full tablet report...
I20260925 04:55:11.637337 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 12757
I20260925 04:55:11.637545 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance
I20260925 04:55:11.638034 6262 external_mini_cluster.cc:1428] Setting key b73621cfd7c8ab8d49f50be1554d0632
Sep 25 04:55:11 dist-test-slave-xzjr krb5kdc[12384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312111, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.29.131@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM
I20260925 04:55:11.653872 12620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:11.635548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:60785 (local address 127.6.29.190:42115)
0925 04:55:11.635825 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:11.635831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:11.637219 (+ 1388us) server_negotiation.cc:407] Connection header received
0925 04:55:11.639507 (+ 2288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:11.639512 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:11.639621 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:11.639744 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:11.641373 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:11.642117 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:11.643756 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:11.644220 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:11.644357 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:11.648114 (+ 3757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:11.648120 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:11.648128 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:11.648162 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:11.650632 (+ 2470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:11.651381 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:11.651390 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:11.651393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:11.651527 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:11.652050 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:11.652062 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:11.652064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:11.652416 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:11.652573 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:11.653310 (+ 737us) server_negotiation.cc:299] Negotiation successful
0925 04:55:11.653524 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 04:55:11.654914 12887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:11.636136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.6.29.131:60785)
0925 04:55:11.637064 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:11.637109 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:11.639205 (+ 2096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:11.640011 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:11.640019 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:11.640534 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:11.641211 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:11.641230 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:11.642281 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:11.642285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:11.643605 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:11.643616 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:11.644443 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:11.644453 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:11.644708 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:11.645368 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:11.645391 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:11.647886 (+ 2495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:11.650772 (+ 2886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:11.650779 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:11.650794 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:11.651247 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:11.651653 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:11.651659 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:11.651662 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:11.651932 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:11.652557 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:11.652565 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:11.652944 (+ 379us) client_negotiation.cc:769] Sending connection context
0925 04:55:11.653222 (+ 278us) client_negotiation.cc:240] Negotiation successful
0925 04:55:11.653793 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":398,"threads_started":1}
I20260925 04:55:11.656299 12885 heartbeater.cc:347] Connected to a master server at 127.6.29.190:42115
I20260925 04:55:11.656636 12885 heartbeater.cc:464] Registering TS with master...
I20260925 04:55:11.657403 12885 heartbeater.cc:511] Master 127.6.29.190:42115 requested a full tablet report, sending...
I20260925 04:55:11.658962 12429 ts_manager.cc:194] Registered new tserver with Master: fbde0dd6885c4cd9ab573d65bfc3231e (127.6.29.131:44161)
I20260925 04:55:11.659898 12429 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.29.131@KRBTEST.COM'} at 127.6.29.131:60785
I20260925 04:55:11.665710 6262 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.6.29.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.29.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.29.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.189.keytab.
Entry for principal oryx/127.6.29.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.189.keytab.
I20260925 04:55:11.716821 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.189:45371
--webserver_interface=127.6.29.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.189.keytab
--principal=oryx/127.6.29.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:44961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:42115,127.6.29.189:45371
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:55:12.051160 12890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:55:12.051460 12890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:55:12.051548 12890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:55:12.058820 12890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:55:12.058936 12890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:55:12.059029 12890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:55:12.059108 12890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:55:12.059192 12890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:55:12.069234 12890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:44961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:42115,127.6.29.189:45371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.189.keytab
--principal=oryx/127.6.29.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.189:45371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.29.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
W20260925 04:55:12.069639 12890 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.6.29.190:42115,127.6.29.189:45371'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 04:55:12.070771 12890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:55:12.072260 12890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:55:12.084789 12896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:12.084707 12898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:12.084681 12895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:55:12.089486 12890 server_base.cc:1034] running on GCE node
I20260925 04:55:12.090493 12890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:55:12.092028 12890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:55:12.093252 12890 hybrid_clock.cc:648] HybridClock initialized: now 1790312112093183 us; error 47 us; skew 500 ppm
I20260925 04:55:12.093554 12890 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:55:12 dist-test-slave-xzjr krb5kdc[12384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312112, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.29.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:55:12.102250 12890 init.cc:377] Logged in from keytab as oryx/127.6.29.189@KRBTEST.COM (short username oryx)
I20260925 04:55:12.104056 12890 webserver.cc:460] Webserver started at http://127.6.29.189:45481/ using document root <none> and password file <none>
I20260925 04:55:12.104867 12890 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:12.104960 12890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:12.105322 12890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:55:12.108696 12890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-1/data/instance:
uuid: "f062a4fd2dc946f99f27578e1c91d17a"
format_stamp: "Formatted at 2026-09-25 04:55:12 on dist-test-slave-xzjr"
server_key: "7578dfc41d1858115b1a0d05c660ae25"
server_key_iv: "7365fdfc5c5787337f7a5aa0ae4aa479"
server_key_version: "encryptionkey@0"
I20260925 04:55:12.109601 12890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-1/wal/instance:
uuid: "f062a4fd2dc946f99f27578e1c91d17a"
format_stamp: "Formatted at 2026-09-25 04:55:12 on dist-test-slave-xzjr"
server_key: "7578dfc41d1858115b1a0d05c660ae25"
server_key_iv: "7365fdfc5c5787337f7a5aa0ae4aa479"
server_key_version: "encryptionkey@0"
I20260925 04:55:12.115202 12890 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260925 04:55:12.119334 12905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:12.120893 12890 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 04:55:12.121083 12890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-1/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-1/wal
uuid: "f062a4fd2dc946f99f27578e1c91d17a"
format_stamp: "Formatted at 2026-09-25 04:55:12 on dist-test-slave-xzjr"
server_key: "7578dfc41d1858115b1a0d05c660ae25"
server_key_iv: "7365fdfc5c5787337f7a5aa0ae4aa479"
server_key_version: "encryptionkey@0"
I20260925 04:55:12.121311 12890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:12.139070 12890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:55:12.143663 12890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:55:12.144018 12890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:55:12.153824 12752 heartbeater.cc:503] Master 127.6.29.190:42115 was elected leader, sending a full tablet report...
I20260925 04:55:12.170411 12890 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.189:45371
I20260925 04:55:12.170433 12956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.189:45371 every 8 connection(s)
I20260925 04:55:12.171943 12890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-1/data/info.pb
I20260925 04:55:12.176184 12957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:55:12.177996 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 12890
I20260925 04:55:12.185989 12957 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 04:55:12 dist-test-slave-xzjr krb5kdc[12384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.29.189@KRBTEST.COM
Sep 25 04:55:12 dist-test-slave-xzjr krb5kdc[12384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312112, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.29.189@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM
I20260925 04:55:12.203419 12960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:12.180361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48804 (local address 127.6.29.189:45371)
0925 04:55:12.181246 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:12.181263 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:12.181326 (+ 63us) server_negotiation.cc:407] Connection header received
0925 04:55:12.183041 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:12.183077 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:12.183626 (+ 549us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:12.184131 (+ 505us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:12.185416 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.187403 (+ 1987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.188882 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.189986 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.190164 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:12.195908 (+ 5744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:12.195915 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:12.195927 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:12.195968 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:12.199051 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:12.199928 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:12.199934 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:12.199941 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:12.200092 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:12.200494 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:12.200497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:12.200499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:12.201135 (+ 636us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:12.201358 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:12.201856 (+ 498us) server_negotiation.cc:299] Negotiation successful
0925 04:55:12.202245 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":414,"threads_started":1}
I20260925 04:55:12.205132 12962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:12.189058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46526 (local address 127.6.29.190:42115)
0925 04:55:12.190651 (+ 1593us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:12.190656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:12.190672 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:55:12.190721 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:12.190724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:12.190783 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:12.190891 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:12.192021 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.192755 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.194602 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.195132 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.195235 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:12.198938 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:12.198943 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:12.198946 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:12.198978 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:12.202287 (+ 3309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:12.203013 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:12.203019 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:12.203023 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:12.203146 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:12.203607 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:12.203615 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:12.203618 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:12.204049 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:12.204197 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:12.204695 (+ 498us) server_negotiation.cc:299] Negotiation successful
0925 04:55:12.204903 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1489,"thread_start_us":1380,"threads_started":1}
I20260925 04:55:12.205428 12961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:12.188980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.0.0.1:46526)
0925 04:55:12.189779 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:12.189822 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:12.189985 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:12.190976 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:12.190990 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:12.191561 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:12.191871 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:12.191887 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.192903 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.192909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:12.194454 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:12.194465 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.195382 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.195391 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:12.195636 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:12.196437 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:12.196464 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:12.198740 (+ 2276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:12.202409 (+ 3669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:12.202421 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:12.202428 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:12.202892 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:12.203253 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:12.203260 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:12.203263 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:12.203443 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:12.204245 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:12.204251 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:12.204383 (+ 132us) client_negotiation.cc:769] Sending connection context
0925 04:55:12.204595 (+ 212us) client_negotiation.cc:240] Negotiation successful
0925 04:55:12.204889 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":349,"threads_started":1}
I20260925 04:55:12.207121 12957 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 04:55:12 dist-test-slave-xzjr krb5kdc[12384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312112, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.29.189@KRBTEST.COM for oryx/127.6.29.189@KRBTEST.COM
I20260925 04:55:12.220386 12961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:12.207854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:45371 (local address 127.0.0.1:48810)
0925 04:55:12.208015 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:12.208030 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:12.208167 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:12.208487 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:12.208491 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:12.208826 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:12.208946 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:12.208954 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.209898 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.209903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:12.210791 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:12.210802 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.211465 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.211470 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:12.211645 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:12.212261 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:12.212286 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:12.214515 (+ 2229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:12.217674 (+ 3159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:12.217679 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:12.217683 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:12.218040 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:12.218386 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:12.218393 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:12.218399 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:12.218538 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:12.219226 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:12.219233 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:12.219400 (+ 167us) client_negotiation.cc:769] Sending connection context
0925 04:55:12.219566 (+ 166us) client_negotiation.cc:240] Negotiation successful
0925 04:55:12.219977 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 04:55:12.220470 12960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:12.208029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48810 (local address 127.6.29.189:45371)
0925 04:55:12.208161 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:12.208166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:12.208183 (+ 17us) server_negotiation.cc:407] Connection header received
0925 04:55:12.208312 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:12.208317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:12.208377 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:12.208473 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:12.209097 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.209763 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.210932 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.211329 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.211460 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:12.214732 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:12.214736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:12.214739 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:12.214766 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:12.217510 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:12.218162 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:12.218168 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:12.218172 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:12.218283 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:12.218691 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:12.218694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:12.218696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:12.219079 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:12.219226 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:12.219805 (+ 579us) server_negotiation.cc:299] Negotiation successful
0925 04:55:12.220141 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":46}
I20260925 04:55:12.226933 12957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a: Bootstrap starting.
I20260925 04:55:12.230221 12957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a: Neither blocks nor log segments found. Creating new log.
I20260925 04:55:12.231251 12957 log.cc:824] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a: Log is configured to *not* fsync() on all Append() calls
I20260925 04:55:12.234004 12957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a: No bootstrap required, opened a new log
I20260925 04:55:12.241465 12957 raft_consensus.cc:348] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } }
I20260925 04:55:12.241791 12957 raft_consensus.cc:374] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:55:12.241868 12957 raft_consensus.cc:729] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f062a4fd2dc946f99f27578e1c91d17a, State: Initialized, Role: FOLLOWER
I20260925 04:55:12.242476 12957 consensus_queue.cc:260] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } }
I20260925 04:55:12.243382 12964 sys_catalog.cc:455] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } } }
I20260925 04:55:12.243712 12964 sys_catalog.cc:458] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:55:12.244557 12957 sys_catalog.cc:564] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [sys.catalog]: configured and running, proceeding with master startup.
W20260925 04:55:12.253851 12975 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:55:12.254025 12975 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:55:12.256598 12890 master_runner.cc:428] Detected that this master f062a4fd2dc946f99f27578e1c91d17a is joining an existing cluster
I20260925 04:55:12.256721 12890 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.29.189:45371
WARNING: no policy specified for oryx/127.6.29.188@KRBTEST.COM; defaulting to no policy
I20260925 04:55:12.288642 12962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:12.262185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46530 (local address 127.6.29.190:42115)
0925 04:55:12.262386 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:12.262392 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:12.269783 (+ 7391us) server_negotiation.cc:407] Connection header received
0925 04:55:12.270898 (+ 1115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:12.270907 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:12.270997 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:12.271085 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:12.272863 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.274066 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.276460 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.277177 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.277260 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:12.281230 (+ 3970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:12.281244 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:12.281251 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:12.281292 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:12.285218 (+ 3926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:12.286274 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:12.286279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:12.286283 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:12.286429 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:12.286885 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:12.286893 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:12.286899 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:12.287309 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:12.287434 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:12.288037 (+ 603us) server_negotiation.cc:299] Negotiation successful
0925 04:55:12.288289 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 04:55:12.289319 12981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:12.262751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.0.0.1:46530)
0925 04:55:12.269597 (+ 6846us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:12.269637 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:12.270704 (+ 1067us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:12.271299 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:12.271303 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:12.272213 (+ 910us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:12.272617 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:12.272631 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.274265 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.274273 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:12.276248 (+ 1975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:12.276262 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.277552 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.277557 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:12.277878 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:12.279726 (+ 1848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:12.279756 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:12.280980 (+ 1224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:12.285378 (+ 4398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:12.285389 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:12.285393 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:12.286120 (+ 727us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:12.286569 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:12.286578 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:12.286583 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:12.286756 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:12.287505 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:12.287509 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:12.287715 (+ 206us) client_negotiation.cc:769] Sending connection context
0925 04:55:12.288493 (+ 778us) client_negotiation.cc:240] Negotiation successful
0925 04:55:12.288842 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6624,"mutex_wait_us":195,"thread_start_us":7066,"threads_started":1}
I20260925 04:55:12.289770 12960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:12.262346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48812 (local address 127.6.29.189:45371)
0925 04:55:12.262589 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:12.262596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:12.270793 (+ 8197us) server_negotiation.cc:407] Connection header received
0925 04:55:12.270974 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:12.270979 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:12.271039 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:12.271109 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:12.273135 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.274213 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.276403 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.277096 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.277196 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:12.281988 (+ 4792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:12.282048 (+ 60us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:12.282054 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:12.282093 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:12.286271 (+ 4178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:12.287086 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:12.287091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:12.287093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:12.287250 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:12.287658 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:12.287663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:12.287666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:12.288034 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:12.288291 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:12.289204 (+ 913us) server_negotiation.cc:299] Negotiation successful
0925 04:55:12.289414 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"spinlock_wait_cycles":4992}
I20260925 04:55:12.290776 12982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:12.262943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:45371 (local address 127.0.0.1:48812)
0925 04:55:12.270698 (+ 7755us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:12.270716 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:12.270873 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:12.271299 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:12.271303 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:12.272605 (+ 1302us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:12.272996 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:12.273007 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.274511 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.274515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:12.276247 (+ 1732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:12.276262 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.277538 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.277545 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:12.277935 (+ 390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:12.279872 (+ 1937us) spinlock_profiling.cc:243] Waited 931 us on lock 0x60c0000347b8. stack: 00007fd7be95b66e 00007fd7be95b430 00007fd7bd9e25ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007fd7be9510fd 00007fd7be950d1e 00007fd7c5468664 00007fd7c5367810 00007fd7c5364470 00007fd7c535f33a 00007fd7c53e6dca 00007fd7c53e4f68 00007fd7c541882f 00007fd7c541858d
0925 04:55:12.280459 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:12.280480 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:12.281751 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:12.286402 (+ 4651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:12.286407 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:12.286410 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:12.286983 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:12.287382 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:12.287387 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:12.287390 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:12.287534 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:12.288354 (+ 820us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:12.288358 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:12.288507 (+ 149us) client_negotiation.cc:769] Sending connection context
0925 04:55:12.290001 (+ 1494us) client_negotiation.cc:240] Negotiation successful
0925 04:55:12.290273 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7623,"mutex_wait_us":164,"spinlock_wait_cycles":2141568,"thread_start_us":7420,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.29.188@KRBTEST.COM" created.
Sep 25 04:55:12 dist-test-slave-xzjr krb5kdc[12384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312109, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.29.190@KRBTEST.COM for oryx/127.6.29.189@KRBTEST.COM
I20260925 04:55:12.317909 12985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:12.299829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:45371 (local address 127.0.0.1:48820)
0925 04:55:12.300817 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:12.300864 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:12.301063 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:12.301557 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:12.301565 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:12.302495 (+ 930us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:12.302808 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:12.302821 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.303980 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.303985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:12.305586 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:12.305601 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.306358 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.306365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:12.306540 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:12.307211 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:12.307233 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:12.310104 (+ 2871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:12.314752 (+ 4648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:12.314772 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:12.314781 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:12.315447 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:12.315955 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:12.315962 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:12.315967 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:12.316198 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:12.316848 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:12.316859 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:12.317060 (+ 201us) client_negotiation.cc:769] Sending connection context
0925 04:55:12.317252 (+ 192us) client_negotiation.cc:240] Negotiation successful
0925 04:55:12.317617 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":544,"threads_started":1}
I20260925 04:55:12.317963 12960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:12.300065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48820 (local address 127.6.29.189:45371)
0925 04:55:12.300261 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:12.300267 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:12.300980 (+ 713us) server_negotiation.cc:407] Connection header received
0925 04:55:12.301288 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:12.301292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:12.301386 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:12.301471 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:12.302980 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.303854 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.305722 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.306191 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.306287 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:12.310321 (+ 4034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:12.310328 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:12.310333 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:12.310375 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:12.314555 (+ 4180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:12.315598 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:12.315605 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:12.315609 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:12.315801 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:12.316323 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:12.316331 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:12.316339 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:12.316676 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:12.316816 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:12.317349 (+ 533us) server_negotiation.cc:299] Negotiation successful
0925 04:55:12.317579 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 04:55:12.320372 12429 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.6.29.189:45371: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a"
member_type: NON_VOTER
last_known_addr {
host: "127.6.29.189"
port: 45371
}
attrs {
promote: true
}
}
dest_uuid: "932891fca8784f3cbae340352149f1fb"
cas_config_opid_index: -1
I20260925 04:55:12.321566 12429 consensus_queue.cc:237] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: NON_VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: true } }
I20260925 04:55:12.325107 12964 raft_consensus.cc:482] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:55:12.325479 12964 raft_consensus.cc:504] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } }
I20260925 04:55:12.325189 12988 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER f062a4fd2dc946f99f27578e1c91d17a (127.6.29.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: NON_VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: true } } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.29.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.188.keytab.
Entry for principal oryx/127.6.29.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.188.keytab.
I20260925 04:55:12.327000 12964 leader_election.cc:288] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 932891fca8784f3cbae340352149f1fb (127.6.29.190:42115)
I20260925 04:55:12.327741 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.188:44375
--webserver_interface=127.6.29.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.188.keytab
--principal=oryx/127.6.29.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:44961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:42115,127.6.29.189:45371,127.6.29.188:44375
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:55:12.328035 12439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f062a4fd2dc946f99f27578e1c91d17a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "932891fca8784f3cbae340352149f1fb" is_pre_election: true
I20260925 04:55:12.329495 12906 leader_election.cc:302] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: f062a4fd2dc946f99f27578e1c91d17a; no voters: 932891fca8784f3cbae340352149f1fb
I20260925 04:55:12.330322 12964 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 04:55:12.331822 12932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:55:12.335875 12932 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [term 1 FOLLOWER]: Refusing update from remote peer 932891fca8784f3cbae340352149f1fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 04:55:12.346916 12991 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: NON_VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:55:12.359273 12988 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.6.29.189:45371
I20260925 04:55:12.360193 12890 master.cc:537] Master@127.6.29.189:45371 shutting down...
I20260925 04:55:12.361318 12987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "932891fca8784f3cbae340352149f1fb" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: NON_VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: true } } }
I20260925 04:55:12.361660 12987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [sys.catalog]: This master's current role is: LEADER
I20260925 04:55:12.401275 12932 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER f062a4fd2dc946f99f27578e1c91d17a (127.6.29.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: NON_VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: true } } }
I20260925 04:55:12.405973 12998 sys_catalog.cc:455] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "932891fca8784f3cbae340352149f1fb" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: NON_VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: true } } }
I20260925 04:55:12.406452 12998 sys_catalog.cc:458] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [sys.catalog]: This master's current role is: LEARNER
I20260925 04:55:12.406407 12964 sys_catalog.cc:455] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 932891fca8784f3cbae340352149f1fb. Latest consensus state: current_term: 1 leader_uuid: "932891fca8784f3cbae340352149f1fb" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: NON_VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: true } } }
I20260925 04:55:12.406630 12964 sys_catalog.cc:458] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [sys.catalog]: This master's current role is: LEARNER
I20260925 04:55:12.406975 12995 sys_catalog.cc:455] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "932891fca8784f3cbae340352149f1fb" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: NON_VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: true } } }
I20260925 04:55:12.407277 12995 sys_catalog.cc:458] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [sys.catalog]: This master's current role is: LEARNER
I20260925 04:55:12.409794 12890 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [term 1 LEARNER]: Raft consensus shutting down.
I20260925 04:55:12.410290 12890 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [term 1 LEARNER]: Raft consensus is shut down!
I20260925 04:55:12.410451 12890 tablet_replica.cc:330] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a: stopping tablet replica
I20260925 04:55:12.418340 12890 master.cc:552] Master@127.6.29.189:45371 shutdown complete.
I20260925 04:55:12.418457 12890 master_runner.cc:305] Clearing existing system tablet
I20260925 04:55:12.418843 12890 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:12.418974 12890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:12.421360 12890 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 04:55:12.422832 13000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:12.423151 12890 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:12.423431 12890 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 04:55:12.423532 12890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-1/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-1/wal
uuid: "f062a4fd2dc946f99f27578e1c91d17a"
format_stamp: "Formatted at 2026-09-25 04:55:12 on dist-test-slave-xzjr"
server_key: "7578dfc41d1858115b1a0d05c660ae25"
server_key_iv: "7365fdfc5c5787337f7a5aa0ae4aa479"
server_key_version: "encryptionkey@0"
I20260925 04:55:12.426542 12890 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:55:12.431475 12890 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 04:55:12.431680 12890 log.cc:1196] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 04:55:12.432201 12890 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a: Deleting consensus metadata
I20260925 04:55:12.432468 12890 master_runner.cc:315] Copying system tablet from 127.6.29.190:42115
I20260925 04:55:12.435029 12890 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a: tablet copy: Beginning tablet copy session from remote peer at address 127.6.29.190:42115
I20260925 04:55:12.448843 12962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:12.435865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46536 (local address 127.6.29.190:42115)
0925 04:55:12.436105 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:12.436113 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:12.436672 (+ 559us) server_negotiation.cc:407] Connection header received
0925 04:55:12.436898 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:12.436904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:12.436982 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:12.437090 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:12.437918 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.438686 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.440183 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.440657 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.440804 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:12.442714 (+ 1910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:12.442720 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:12.442725 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:12.442770 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:12.445826 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:12.446628 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:12.446634 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:12.446637 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:12.446772 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:12.447228 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:12.447233 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:12.447236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:12.447573 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:12.447795 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:12.448231 (+ 436us) server_negotiation.cc:299] Negotiation successful
0925 04:55:12.448440 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 04:55:12.448915 13006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:12.435757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.0.0.1:46536)
0925 04:55:12.436556 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:12.436572 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:12.436749 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:12.437116 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:12.437121 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:12.437522 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:12.437751 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:12.437760 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.438835 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.438840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:12.440031 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:12.440043 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.440822 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.440828 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:12.441054 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:12.441741 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:12.441759 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:12.442521 (+ 762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:12.445958 (+ 3437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:12.445963 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:12.445967 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:12.446491 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:12.446884 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:12.446889 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:12.446893 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:12.447052 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:12.447741 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:12.447745 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:12.447951 (+ 206us) client_negotiation.cc:769] Sending connection context
0925 04:55:12.448127 (+ 176us) client_negotiation.cc:240] Negotiation successful
0925 04:55:12.448439 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":499,"threads_started":1}
I20260925 04:55:12.449870 12449 tablet_copy_service.cc:140] P 932891fca8784f3cbae340352149f1fb: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f062a4fd2dc946f99f27578e1c91d17a ({username='oryx', principal='oryx/127.6.29.189@KRBTEST.COM'} at 127.0.0.1:46536)
I20260925 04:55:12.450176 12449 tablet_copy_service.cc:161] P 932891fca8784f3cbae340352149f1fb: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f062a4fd2dc946f99f27578e1c91d17a at {username='oryx', principal='oryx/127.6.29.189@KRBTEST.COM'} at 127.0.0.1:46536: session id = f062a4fd2dc946f99f27578e1c91d17a-00000000000000000000000000000000
I20260925 04:55:12.455247 12449 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 04:55:12.457778 12890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 04:55:12.465708 12890 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a: tablet copy: Starting download of 0 data blocks...
I20260925 04:55:12.466086 12890 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a: tablet copy: Starting download of 1 WAL segments...
I20260925 04:55:12.469301 12890 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 04:55:12.474874 12449 tablet_copy_service.cc:342] P 932891fca8784f3cbae340352149f1fb: Request end of tablet copy session f062a4fd2dc946f99f27578e1c91d17a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.29.189@KRBTEST.COM'} at 127.0.0.1:46536
I20260925 04:55:12.475087 12449 tablet_copy_service.cc:433] P 932891fca8784f3cbae340352149f1fb: ending tablet copy session f062a4fd2dc946f99f27578e1c91d17a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f062a4fd2dc946f99f27578e1c91d17a
I20260925 04:55:12.489387 12890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:55:12.492430 13007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:12.493911 13008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:12.498420 13010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:55:12.501277 12890 server_base.cc:1034] running on GCE node
I20260925 04:55:12.501669 12890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:55:12.502390 12890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:55:12.503607 12890 hybrid_clock.cc:648] HybridClock initialized: now 1790312112503550 us; error 33 us; skew 500 ppm
I20260925 04:55:12.503904 12890 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:55:12 dist-test-slave-xzjr krb5kdc[12384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312112, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.29.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:55:12.506134 12890 init.cc:377] Logged in from keytab as oryx/127.6.29.189@KRBTEST.COM (short username oryx)
I20260925 04:55:12.508744 12890 webserver.cc:460] Webserver started at http://127.6.29.189:44639/ using document root <none> and password file <none>
I20260925 04:55:12.509119 12890 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:12.509200 12890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:12.512988 12890 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260925 04:55:12.515079 13017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:12.515563 12890 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 04:55:12.515745 12890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-1/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-1/wal
uuid: "f062a4fd2dc946f99f27578e1c91d17a"
format_stamp: "Formatted at 2026-09-25 04:55:12 on dist-test-slave-xzjr"
server_key: "7578dfc41d1858115b1a0d05c660ae25"
server_key_iv: "7365fdfc5c5787337f7a5aa0ae4aa479"
server_key_version: "encryptionkey@0"
I20260925 04:55:12.516027 12890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:12.550575 12890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:55:12.555140 12890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:55:12.577744 12890 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.189:45371
I20260925 04:55:12.577771 13068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.189:45371 every 8 connection(s)
I20260925 04:55:12.578950 12890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-1/data/info.pb
I20260925 04:55:12.584744 13069 sys_catalog.cc:263] Verifying existing consensus state
I20260925 04:55:12.588330 13069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a: Bootstrap starting.
I20260925 04:55:12.604503 13069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:55:12.605054 13069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a: Bootstrap complete.
I20260925 04:55:12.606110 13069 raft_consensus.cc:348] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: NON_VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: true } }
I20260925 04:55:12.606365 13069 raft_consensus.cc:729] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f062a4fd2dc946f99f27578e1c91d17a, State: Initialized, Role: LEARNER
I20260925 04:55:12.606603 13069 consensus_queue.cc:260] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: NON_VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: true } }
I20260925 04:55:12.607496 13072 sys_catalog.cc:455] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: NON_VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: true } } }
I20260925 04:55:12.607838 13069 sys_catalog.cc:564] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:55:12.607993 13072 sys_catalog.cc:458] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [sys.catalog]: This master's current role is: LEARNER
I20260925 04:55:12.615458 13083 catalog_manager.cc:1259] Loaded cluster ID: f48087faeea44abca781b012f0655bbc
I20260925 04:55:12.615571 13083 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a: loading cluster ID for follower catalog manager: success
I20260925 04:55:12.619483 13083 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a: acquiring CA information for follower catalog manager: success
I20260925 04:55:12.622113 13083 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 04:55:12.662566 12885 heartbeater.cc:503] Master 127.6.29.190:42115 was elected leader, sending a full tablet report...
W20260925 04:55:12.746786 12990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:55:12.747139 12990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:55:12.747303 12990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:55:12.754734 12990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:55:12.754851 12990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:55:12.754927 12990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:55:12.755016 12990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:55:12.755105 12990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:55:12.765553 12990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:44961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:42115,127.6.29.189:45371,127.6.29.188:44375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.188.keytab
--principal=oryx/127.6.29.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.188:44375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.29.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:55:12.766970 12990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:55:12.768361 12990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:55:12.780086 13088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:12.780154 13089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:12.780063 13091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:55:12.781141 12990 server_base.cc:1034] running on GCE node
I20260925 04:55:12.781921 12990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:55:12.783363 12990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:55:12.784554 12990 hybrid_clock.cc:648] HybridClock initialized: now 1790312112784495 us; error 58 us; skew 500 ppm
I20260925 04:55:12.784930 12990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:55:12 dist-test-slave-xzjr krb5kdc[12384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312112, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.29.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:55:12.788120 12990 init.cc:377] Logged in from keytab as oryx/127.6.29.188@KRBTEST.COM (short username oryx)
I20260925 04:55:12.789652 12990 webserver.cc:460] Webserver started at http://127.6.29.188:42671/ using document root <none> and password file <none>
I20260925 04:55:12.790356 12990 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:12.790501 12990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:12.790813 12990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:55:12.793977 12990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-2/data/instance:
uuid: "15b0f62dff3a440587c04fb55216d081"
format_stamp: "Formatted at 2026-09-25 04:55:12 on dist-test-slave-xzjr"
server_key: "9600a550ce83c76d9af74cb61a0eba46"
server_key_iv: "f8c5b3a1786a9972618c5539ed904270"
server_key_version: "encryptionkey@0"
I20260925 04:55:12.794806 12990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-2/wal/instance:
uuid: "15b0f62dff3a440587c04fb55216d081"
format_stamp: "Formatted at 2026-09-25 04:55:12 on dist-test-slave-xzjr"
server_key: "9600a550ce83c76d9af74cb61a0eba46"
server_key_iv: "f8c5b3a1786a9972618c5539ed904270"
server_key_version: "encryptionkey@0"
I20260925 04:55:12.800144 12990 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 04:55:12.803960 13098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:12.805549 12990 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:55:12.805706 12990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-2/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-2/wal
uuid: "15b0f62dff3a440587c04fb55216d081"
format_stamp: "Formatted at 2026-09-25 04:55:12 on dist-test-slave-xzjr"
server_key: "9600a550ce83c76d9af74cb61a0eba46"
server_key_iv: "f8c5b3a1786a9972618c5539ed904270"
server_key_version: "encryptionkey@0"
I20260925 04:55:12.805881 12990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:12.826932 12990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:55:12.831522 12990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:55:12.831915 12990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:55:12.850216 12990 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.188:44375
I20260925 04:55:12.850250 13149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.188:44375 every 8 connection(s)
I20260925 04:55:12.851660 12990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-2/data/info.pb
I20260925 04:55:12.855743 13150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:55:12.859586 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 12990
I20260925 04:55:12.864014 13150 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 04:55:12 dist-test-slave-xzjr krb5kdc[12384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.29.188@KRBTEST.COM
Sep 25 04:55:12 dist-test-slave-xzjr krb5kdc[12384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312112, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.29.188@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM
I20260925 04:55:12.882185 12962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:12.867672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46538 (local address 127.6.29.190:42115)
0925 04:55:12.867889 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:12.867895 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:12.868402 (+ 507us) server_negotiation.cc:407] Connection header received
0925 04:55:12.868564 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:12.868570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:12.868651 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:12.868754 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:12.869794 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.870466 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.871921 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.872478 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.872621 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:12.876676 (+ 4055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:12.876680 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:12.876684 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:12.876716 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:12.879252 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:12.880068 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:12.880075 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:12.880083 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:12.880220 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:12.880630 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:12.880634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:12.880636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:12.880891 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:12.881040 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:12.881585 (+ 545us) server_negotiation.cc:299] Negotiation successful
0925 04:55:12.881816 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 04:55:12.883376 13154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:12.867518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.0.0.1:46538)
0925 04:55:12.868255 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:12.868295 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:12.868449 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:12.868805 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:12.868809 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:12.869412 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:12.869650 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:12.869658 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.870607 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.870614 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:12.871698 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:12.871713 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.872703 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.872708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:12.872826 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:12.873454 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:12.873486 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:12.876472 (+ 2986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:12.879432 (+ 2960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:12.879442 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:12.879456 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:12.879938 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:12.880332 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:12.880338 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:12.880341 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:12.880504 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:12.881017 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:12.881025 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:12.881237 (+ 212us) client_negotiation.cc:769] Sending connection context
0925 04:55:12.881505 (+ 268us) client_negotiation.cc:240] Negotiation successful
0925 04:55:12.882068 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"spinlock_wait_cycles":21376,"thread_start_us":326,"threads_started":1}
I20260925 04:55:12.883394 13153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:12.861912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57516 (local address 127.6.29.188:44375)
0925 04:55:12.862651 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:12.862666 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:12.862715 (+ 49us) server_negotiation.cc:407] Connection header received
0925 04:55:12.864804 (+ 2089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:12.864837 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:12.865417 (+ 580us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:12.865889 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:12.867310 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.869164 (+ 1854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.870411 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.871092 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.871282 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:12.876260 (+ 4978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:12.876270 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:12.876287 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:12.876328 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:12.879254 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:12.880127 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:12.880138 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:12.880142 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:12.880303 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:12.880704 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:12.880710 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:12.880715 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:12.881233 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:12.881550 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:12.881840 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 04:55:12.882229 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":110,"server-negotiator.queue_time_us":507,"thread_start_us":299,"threads_started":1}
I20260925 04:55:12.885795 13150 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 04:55:12 dist-test-slave-xzjr krb5kdc[12384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312112, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.29.188@KRBTEST.COM for oryx/127.6.29.189@KRBTEST.COM
I20260925 04:55:12.899052 13154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:12.886644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:45371 (local address 127.0.0.1:48834)
0925 04:55:12.886804 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:12.886835 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:12.886948 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:12.887459 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:12.887463 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:12.887877 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:12.888009 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:12.888017 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.889259 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.889265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:12.890177 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:12.890188 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.890874 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.890880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:12.891065 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:12.891620 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:12.891637 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:12.893628 (+ 1991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:12.896616 (+ 2988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:12.896624 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:12.896630 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:12.897015 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:12.897413 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:12.897418 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:12.897422 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:12.897559 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:12.898069 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:12.898075 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:12.898233 (+ 158us) client_negotiation.cc:769] Sending connection context
0925 04:55:12.898371 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 04:55:12.898696 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 04:55:12.899063 13155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:12.886657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48834 (local address 127.6.29.189:45371)
0925 04:55:12.887200 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:12.887205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:12.887219 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:55:12.887266 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:12.887269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:12.887323 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:12.887440 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:12.888152 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.889122 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.890312 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.890723 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.890856 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:12.893836 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:12.893840 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:12.893844 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:12.893876 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:12.896475 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:12.897156 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:12.897161 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:12.897167 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:12.897299 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:12.897678 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:12.897684 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:12.897687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:12.897952 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:12.898116 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:12.898466 (+ 350us) server_negotiation.cc:299] Negotiation successful
0925 04:55:12.898696 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":268,"threads_started":1}
I20260925 04:55:12.900345 13150 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.29.188" port: 44375 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 04:55:12 dist-test-slave-xzjr krb5kdc[12384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312112, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.29.188@KRBTEST.COM for oryx/127.6.29.188@KRBTEST.COM
I20260925 04:55:12.913079 13154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:12.901099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:44375 (local address 127.0.0.1:57518)
0925 04:55:12.901353 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:12.901367 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:12.901451 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:12.901699 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:12.901702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:12.901999 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:12.902146 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:12.902153 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.903033 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.903037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:12.903975 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:12.903985 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.904668 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.904672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:12.904834 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:12.905423 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:12.905441 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:12.907490 (+ 2049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:12.910821 (+ 3331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:12.910824 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:12.910827 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:12.911211 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:12.911553 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:12.911557 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:12.911559 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:12.911707 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:12.912314 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:12.912318 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:12.912448 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 04:55:12.912591 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 04:55:12.912790 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260925 04:55:12.913209 13153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:12.901099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57518 (local address 127.6.29.188:44375)
0925 04:55:12.901256 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:12.901261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:12.901433 (+ 172us) server_negotiation.cc:407] Connection header received
0925 04:55:12.901523 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:12.901528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:12.901586 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:12.901667 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:12.902257 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.902926 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.904091 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.904499 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.904646 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:12.907680 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:12.907726 (+ 46us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:12.907733 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:12.907805 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:12.910705 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:12.911305 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:12.911309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:12.911311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:12.911423 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:12.911871 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:12.911875 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:12.911877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:12.912182 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:12.912294 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:12.912699 (+ 405us) server_negotiation.cc:299] Negotiation successful
0925 04:55:12.912902 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":58}
I20260925 04:55:12.918182 13150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081: Bootstrap starting.
I20260925 04:55:12.921366 13150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081: Neither blocks nor log segments found. Creating new log.
I20260925 04:55:12.922430 13150 log.cc:824] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081: Log is configured to *not* fsync() on all Append() calls
I20260925 04:55:12.924261 13157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "932891fca8784f3cbae340352149f1fb" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: NON_VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: true } } }
I20260925 04:55:12.924532 13157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [sys.catalog]: This master's current role is: LEADER
I20260925 04:55:12.925940 13150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081: No bootstrap required, opened a new log
I20260925 04:55:12.934526 13155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:12.925069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48846 (local address 127.6.29.189:45371)
0925 04:55:12.925307 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:12.925316 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:12.928176 (+ 2860us) server_negotiation.cc:407] Connection header received
0925 04:55:12.928886 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:12.928892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:12.929001 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:12.929079 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:55:12.929770 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.930455 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.932847 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.933641 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.933781 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:12.933973 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:12.934256 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 04:55:12.934355 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 04:55:12.934716 13160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:12.924486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:45371 (local address 127.0.0.1:48846)
0925 04:55:12.928052 (+ 3566us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:12.928072 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:12.928697 (+ 625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:12.929312 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:12.929317 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:12.929344 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:55:12.929579 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:12.929589 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.930593 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.930598 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:12.932686 (+ 2088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:12.932702 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.933848 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.933854 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:12.934053 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:12.934131 (+ 78us) client_negotiation.cc:769] Sending connection context
0925 04:55:12.934235 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 04:55:12.934378 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3419,"thread_start_us":3262,"threads_started":1}
I20260925 04:55:12.935716 13072 sys_catalog.cc:455] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 932891fca8784f3cbae340352149f1fb. Latest consensus state: current_term: 1 leader_uuid: "932891fca8784f3cbae340352149f1fb" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: NON_VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: true } } }
I20260925 04:55:12.936003 13072 sys_catalog.cc:458] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [sys.catalog]: This master's current role is: LEARNER
I20260925 04:55:12.936230 13150 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } } peers { permanent_uuid: "15b0f62dff3a440587c04fb55216d081" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 44375 } }
I20260925 04:55:12.936573 13159 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb: attempting to promote NON_VOTER f062a4fd2dc946f99f27578e1c91d17a to VOTER
I20260925 04:55:12.936661 13150 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:55:12.936787 13150 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15b0f62dff3a440587c04fb55216d081, State: Initialized, Role: FOLLOWER
I20260925 04:55:12.937462 13159 consensus_queue.cc:237] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: false } }
I20260925 04:55:12.937453 13150 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } } peers { permanent_uuid: "15b0f62dff3a440587c04fb55216d081" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 44375 } }
I20260925 04:55:12.938409 13162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } } peers { permanent_uuid: "15b0f62dff3a440587c04fb55216d081" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 44375 } } }
I20260925 04:55:12.938853 13162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:55:12.939353 13044 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [term 1 LEARNER]: Refusing update from remote peer 932891fca8784f3cbae340352149f1fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 04:55:12.939800 13150 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:55:12.939929 13158 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 04:55:12.943089 13159 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, f062a4fd2dc946f99f27578e1c91d17a (127.6.29.189) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: false } } }
I20260925 04:55:12.943866 13044 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, f062a4fd2dc946f99f27578e1c91d17a (127.6.29.189) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: false } } }
I20260925 04:55:12.946152 13159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "932891fca8784f3cbae340352149f1fb" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: false } } }
I20260925 04:55:12.946385 13159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [sys.catalog]: This master's current role is: LEADER
I20260925 04:55:12.946713 13157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "932891fca8784f3cbae340352149f1fb" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: false } } }
I20260925 04:55:12.946878 13157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [sys.catalog]: This master's current role is: LEADER
I20260925 04:55:12.949084 13072 sys_catalog.cc:455] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "932891fca8784f3cbae340352149f1fb" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: false } } }
I20260925 04:55:12.949344 13072 sys_catalog.cc:458] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [sys.catalog]: This master's current role is: FOLLOWER
W20260925 04:55:12.954193 13174 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:55:12.954331 13174 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:55:12.958093 12990 master_runner.cc:428] Detected that this master 15b0f62dff3a440587c04fb55216d081 is joining an existing cluster
I20260925 04:55:12.958233 12990 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.29.188:44375
I20260925 04:55:12.981413 12962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:12.962699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46550 (local address 127.6.29.190:42115)
0925 04:55:12.962890 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:12.962898 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:12.963853 (+ 955us) server_negotiation.cc:407] Connection header received
0925 04:55:12.964346 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:12.964356 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:12.964479 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:12.964582 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:12.966464 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.967637 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.969654 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.970322 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.970422 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:12.973260 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:12.973266 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:12.973271 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:12.973310 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:12.978206 (+ 4896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:12.979000 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:12.979006 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:12.979009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:12.979161 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:12.979647 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:12.979652 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:12.979655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:12.980021 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:12.980173 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:12.980821 (+ 648us) server_negotiation.cc:299] Negotiation successful
0925 04:55:12.981053 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 04:55:12.982447 13182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:12.962183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.0.0.1:46550)
0925 04:55:12.963695 (+ 1512us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:12.963733 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:12.964011 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:12.964884 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:12.964890 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:12.965822 (+ 932us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:12.966302 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:12.966313 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.967851 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.967856 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:12.969492 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:12.969505 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.970692 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.970698 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:12.970890 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:12.971833 (+ 943us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:12.971857 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:12.973055 (+ 1198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:12.978361 (+ 5306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:12.978367 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:12.978371 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:12.978860 (+ 489us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:12.979310 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:12.979314 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:12.979318 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:12.979513 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:12.980252 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:12.980258 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:12.980467 (+ 209us) client_negotiation.cc:769] Sending connection context
0925 04:55:12.981651 (+ 1184us) client_negotiation.cc:240] Negotiation successful
0925 04:55:12.981984 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1372,"thread_start_us":12158,"threads_started":1}
I20260925 04:55:12.989976 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 12490
I20260925 04:55:12.992360 13153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:12.963053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57522 (local address 127.6.29.188:44375)
0925 04:55:12.963208 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:12.963217 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:12.974976 (+ 11759us) server_negotiation.cc:407] Connection header received
0925 04:55:12.975164 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:12.975168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:12.975231 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:12.975296 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:12.978405 (+ 3109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.979489 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.981345 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.982015 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.982102 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:12.985040 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:12.985046 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:12.985051 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:12.985085 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:12.989240 (+ 4155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:12.989967 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:12.989973 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:12.989977 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:12.990104 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:12.990437 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:12.990449 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:12.990453 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:12.990762 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:12.990890 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:12.991483 (+ 593us) server_negotiation.cc:299] Negotiation successful
0925 04:55:12.991810 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":12,"server-negotiator.queue_time_us":32}
I20260925 04:55:12.994369 13181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:12.963016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:44375 (local address 127.0.0.1:57522)
0925 04:55:12.974898 (+ 11882us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:12.974915 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:12.975048 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:12.975471 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:12.975476 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:12.977971 (+ 2495us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:12.978261 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:12.978273 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.979704 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.979709 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:12.981217 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:12.981229 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.982803 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.982812 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:12.982994 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:12.983688 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:12.983711 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:12.984847 (+ 1136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:12.989386 (+ 4539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:12.989391 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:12.989395 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:12.989867 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:12.990225 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:12.990229 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:12.990233 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:12.990353 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:12.990960 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:12.990965 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:12.991153 (+ 188us) client_negotiation.cc:769] Sending connection context
0925 04:55:12.993254 (+ 2101us) client_negotiation.cc:240] Negotiation successful
0925 04:55:12.993679 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11755,"thread_start_us":395,"threads_started":1}
Sep 25 04:55:13 dist-test-slave-xzjr krb5kdc[12384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312109, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.29.190@KRBTEST.COM for oryx/127.6.29.188@KRBTEST.COM
I20260925 04:55:13.007225 13155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:12.962332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48860 (local address 127.6.29.189:45371)
0925 04:55:12.962536 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:12.962544 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:12.974795 (+ 12251us) server_negotiation.cc:407] Connection header received
0925 04:55:12.975062 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:12.975070 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:12.975179 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:12.975251 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:12.977698 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.978844 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.980900 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.981566 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.981670 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:12.987453 (+ 5783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:12.987462 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:12.987469 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:12.987518 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:12.990938 (+ 3420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:13.000089 (+ 9151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:13.000099 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:13.000105 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:13.000347 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:13.003314 (+ 2967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:13.003323 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:13.003329 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:13.003866 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:13.004063 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:13.006524 (+ 2461us) server_negotiation.cc:299] Negotiation successful
0925 04:55:13.006870 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 04:55:13.011226 13180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:12.962322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:45371 (local address 127.0.0.1:48860)
0925 04:55:12.974589 (+ 12267us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:12.974681 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:12.974908 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:12.975471 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:12.975476 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:12.977181 (+ 1705us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:12.977539 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:12.977552 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.979010 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.979016 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:12.980741 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:12.980754 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.981968 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.981974 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:12.985252 (+ 3278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:12.986003 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:12.986031 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:12.987207 (+ 1176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:12.991115 (+ 3908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:12.991120 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:12.991124 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:12.999879 (+ 8755us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:13.000531 (+ 652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:13.000539 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:13.000546 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:13.003124 (+ 2578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:13.005247 (+ 2123us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:13.005254 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:13.005600 (+ 346us) client_negotiation.cc:769] Sending connection context
0925 04:55:13.005897 (+ 297us) client_negotiation.cc:240] Negotiation successful
0925 04:55:13.007528 (+ 1631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12086,"mutex_wait_us":3058,"thread_start_us":13440,"threads_started":1}
I20260925 04:55:13.017338 13160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:12.988209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:44375 (local address 127.0.0.1:57532)
0925 04:55:12.988842 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:12.988861 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:12.988965 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:12.993098 (+ 4133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:12.993103 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:12.993595 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:12.993798 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:12.993812 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.996899 (+ 3087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.996908 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:12.998560 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:12.998578 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:13.000079 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:13.000090 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:13.000362 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:13.001220 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:13.001254 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:13.005004 (+ 3750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:13.014178 (+ 9174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:13.014207 (+ 29us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:13.014211 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:13.014823 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:13.015284 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:13.015289 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:13.015293 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:13.015490 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:13.016252 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:13.016258 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:13.016483 (+ 225us) client_negotiation.cc:769] Sending connection context
0925 04:55:13.016645 (+ 162us) client_negotiation.cc:240] Negotiation successful
0925 04:55:13.016926 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494}
I20260925 04:55:13.017716 13153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:12.989182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57532 (local address 127.6.29.188:44375)
0925 04:55:12.992646 (+ 3464us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:12.992654 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:12.992679 (+ 25us) server_negotiation.cc:407] Connection header received
0925 04:55:12.992790 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:12.992798 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:12.992936 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:12.993069 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:12.993990 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.996741 (+ 2751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:12.998785 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:12.999851 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:13.000796 (+ 945us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:13.005255 (+ 4459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:13.005264 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:13.005272 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:13.005324 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:13.013991 (+ 8667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:13.014965 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:13.014971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:13.014975 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:13.015136 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:13.015626 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:13.015632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:13.015641 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:13.016029 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:13.016249 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:13.016753 (+ 504us) server_negotiation.cc:299] Negotiation successful
0925 04:55:13.016991 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3368,"thread_start_us":21980,"threads_started":1}
I20260925 04:55:13.019448 12429 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.6.29.188:44375: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "15b0f62dff3a440587c04fb55216d081"
member_type: NON_VOTER
last_known_addr {
host: "127.6.29.188"
port: 44375
}
attrs {
promote: true
}
}
dest_uuid: "932891fca8784f3cbae340352149f1fb"
cas_config_opid_index: 6
I20260925 04:55:13.020282 12429 consensus_queue.cc:237] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: false } } peers { permanent_uuid: "15b0f62dff3a440587c04fb55216d081" member_type: NON_VOTER last_known_addr { host: "127.6.29.188" port: 44375 } attrs { promote: true } }
Sep 25 04:55:13 dist-test-slave-xzjr krb5kdc[12384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.29.129@KRBTEST.COM
I20260925 04:55:13.022804 13044 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [term 1 FOLLOWER]: Refusing update from remote peer 932891fca8784f3cbae340352149f1fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 04:55:13.023425 13159 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 04:55:13.023782 13125 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:55:13.026254 13159 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 15b0f62dff3a440587c04fb55216d081 (127.6.29.188) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: false } } peers { permanent_uuid: "15b0f62dff3a440587c04fb55216d081" member_type: NON_VOTER last_known_addr { host: "127.6.29.188" port: 44375 } attrs { promote: true } } }
I20260925 04:55:13.027907 13125 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [term 1 FOLLOWER]: Refusing update from remote peer 932891fca8784f3cbae340352149f1fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 04:55:13.030172 13158 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "15b0f62dff3a440587c04fb55216d081" member_type: NON_VOTER last_known_addr { host: "127.6.29.188" port: 44375 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:55:13.031241 13184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:12.991101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54676 (local address 127.6.29.129:46555)
0925 04:55:13.006909 (+ 15808us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:13.006919 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:13.006951 (+ 32us) server_negotiation.cc:407] Connection header received
0925 04:55:13.007019 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:13.007028 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:13.007361 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:13.011945 (+ 4584us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:13.012239 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:13.013961 (+ 1722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:13.015896 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:13.016905 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:13.017861 (+ 956us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:13.022368 (+ 4507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:13.022381 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:13.022397 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:13.022463 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:13.027514 (+ 5051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:13.028771 (+ 1257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:13.028780 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:13.028784 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:13.028951 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:13.029467 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:13.029470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:13.029472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:13.029820 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:13.030135 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:13.030666 (+ 531us) server_negotiation.cc:299] Negotiation successful
0925 04:55:13.030936 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15619,"thread_start_us":15436,"threads_started":1}
I20260925 04:55:13.032302 13159 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.6.29.188:44375
I20260925 04:55:13.032912 12990 master.cc:537] Master@127.6.29.188:44375 shutting down...
I20260925 04:55:13.032733 13044 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 15b0f62dff3a440587c04fb55216d081 (127.6.29.188) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: false } } peers { permanent_uuid: "15b0f62dff3a440587c04fb55216d081" member_type: NON_VOTER last_known_addr { host: "127.6.29.188" port: 44375 } attrs { promote: true } } }
I20260925 04:55:13.033783 13159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "932891fca8784f3cbae340352149f1fb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: false } } peers { permanent_uuid: "15b0f62dff3a440587c04fb55216d081" member_type: NON_VOTER last_known_addr { host: "127.6.29.188" port: 44375 } attrs { promote: true } } }
I20260925 04:55:13.034090 13159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [sys.catalog]: This master's current role is: LEADER
I20260925 04:55:13.034373 13157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "932891fca8784f3cbae340352149f1fb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: false } } peers { permanent_uuid: "15b0f62dff3a440587c04fb55216d081" member_type: NON_VOTER last_known_addr { host: "127.6.29.188" port: 44375 } attrs { promote: true } } }
I20260925 04:55:13.034613 13157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [sys.catalog]: This master's current role is: LEADER
I20260925 04:55:13.036221 12613 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:55:13.038242 13072 sys_catalog.cc:455] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "932891fca8784f3cbae340352149f1fb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: false } } peers { permanent_uuid: "15b0f62dff3a440587c04fb55216d081" member_type: NON_VOTER last_known_addr { host: "127.6.29.188" port: 44375 } attrs { promote: true } } }
I20260925 04:55:13.038543 13072 sys_catalog.cc:458] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [sys.catalog]: This master's current role is: FOLLOWER
W20260925 04:55:13.041486 12415 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260925 04:55:13.048214 12415 consensus_peers.cc:487] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb -> Peer 15b0f62dff3a440587c04fb55216d081 (127.6.29.188:44375): Couldn't send request to peer 15b0f62dff3a440587c04fb55216d081. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 04:55:13.066253 12990 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:55:13.066751 12990 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:55:13.066967 12990 tablet_replica.cc:330] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081: stopping tablet replica
I20260925 04:55:13.084115 12990 master.cc:552] Master@127.6.29.188:44375 shutdown complete.
I20260925 04:55:13.084226 12990 master_runner.cc:305] Clearing existing system tablet
I20260925 04:55:13.084614 12990 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:13.084728 12990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:13.087752 12990 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260925 04:55:13.089507 13188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:13.089870 12990 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:13.090127 12990 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 04:55:13.090224 12990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-2/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-2/wal
uuid: "15b0f62dff3a440587c04fb55216d081"
format_stamp: "Formatted at 2026-09-25 04:55:12 on dist-test-slave-xzjr"
server_key: "9600a550ce83c76d9af74cb61a0eba46"
server_key_iv: "f8c5b3a1786a9972618c5539ed904270"
server_key_version: "encryptionkey@0"
I20260925 04:55:13.095404 12990 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:55:13.102698 12990 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 04:55:13.102885 12990 log.cc:1196] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 04:55:13.103498 12990 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081: Deleting consensus metadata
I20260925 04:55:13.103756 12990 master_runner.cc:315] Copying system tablet from 127.6.29.190:42115
I20260925 04:55:13.109627 12990 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081: tablet copy: Beginning tablet copy session from remote peer at address 127.6.29.190:42115
I20260925 04:55:13.126230 12962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:13.110610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46562 (local address 127.6.29.190:42115)
0925 04:55:13.110826 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:13.110832 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:13.111424 (+ 592us) server_negotiation.cc:407] Connection header received
0925 04:55:13.111612 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:13.111618 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:13.111707 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:13.111842 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:13.112843 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:13.114009 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:13.115960 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:13.116683 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:13.116867 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:13.119446 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:13.119453 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:13.119458 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:13.119497 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:13.123185 (+ 3688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:13.124077 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:13.124083 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:13.124086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:13.124235 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:13.124678 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:13.124682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:13.124685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:13.125015 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:13.125165 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:13.125642 (+ 477us) server_negotiation.cc:299] Negotiation successful
0925 04:55:13.125876 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260925 04:55:13.126667 13194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:13.110542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.0.0.1:46562)
0925 04:55:13.111286 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:13.111305 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:13.111471 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:13.111870 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:13.111875 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:13.112406 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:13.112676 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:13.112690 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:13.114188 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:13.114193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:13.115810 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:13.115827 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:13.116910 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:13.116916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:13.117102 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:13.118020 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:13.118043 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:13.119216 (+ 1173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:13.123351 (+ 4135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:13.123358 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:13.123363 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:13.123945 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:13.124355 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:13.124365 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:13.124369 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:13.124562 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:13.125150 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:13.125156 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:13.125383 (+ 227us) client_negotiation.cc:769] Sending connection context
0925 04:55:13.125536 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 04:55:13.125909 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":456,"threads_started":1}
I20260925 04:55:13.127414 12449 tablet_copy_service.cc:140] P 932891fca8784f3cbae340352149f1fb: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 15b0f62dff3a440587c04fb55216d081 ({username='oryx', principal='oryx/127.6.29.188@KRBTEST.COM'} at 127.0.0.1:46562)
I20260925 04:55:13.127611 12449 tablet_copy_service.cc:161] P 932891fca8784f3cbae340352149f1fb: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 15b0f62dff3a440587c04fb55216d081 at {username='oryx', principal='oryx/127.6.29.188@KRBTEST.COM'} at 127.0.0.1:46562: session id = 15b0f62dff3a440587c04fb55216d081-00000000000000000000000000000000
I20260925 04:55:13.131670 12449 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 04:55:13.133826 12990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 04:55:13.141433 12990 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081: tablet copy: Starting download of 0 data blocks...
I20260925 04:55:13.141798 12990 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081: tablet copy: Starting download of 1 WAL segments...
I20260925 04:55:13.144665 12990 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 04:55:13.149552 12449 tablet_copy_service.cc:342] P 932891fca8784f3cbae340352149f1fb: Request end of tablet copy session 15b0f62dff3a440587c04fb55216d081-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.29.188@KRBTEST.COM'} at 127.0.0.1:46562
I20260925 04:55:13.149709 12449 tablet_copy_service.cc:433] P 932891fca8784f3cbae340352149f1fb: ending tablet copy session 15b0f62dff3a440587c04fb55216d081-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 15b0f62dff3a440587c04fb55216d081
I20260925 04:55:13.160833 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 12490
I20260925 04:55:13.164896 12990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:55:13.169504 13195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:13.169878 13196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:13.182024 13198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:55:13.182529 12990 server_base.cc:1034] running on GCE node
I20260925 04:55:13.182981 12990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:55:13.183717 12990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:55:13.184937 12990 hybrid_clock.cc:648] HybridClock initialized: now 1790312113184894 us; error 37 us; skew 500 ppm
I20260925 04:55:13.185184 12990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:55:13 dist-test-slave-xzjr krb5kdc[12384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312113, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.29.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:55:13.187161 12990 init.cc:377] Logged in from keytab as oryx/127.6.29.188@KRBTEST.COM (short username oryx)
I20260925 04:55:13.188817 12990 webserver.cc:460] Webserver started at http://127.6.29.188:46177/ using document root <none> and password file <none>
I20260925 04:55:13.189165 12990 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:13.189106 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.129:46555
--local_ip_for_outbound_sockets=127.6.29.129
--tserver_master_addrs=127.6.29.190:42115
--webserver_port=33697
--webserver_interface=127.6.29.129
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.129.keytab
--principal=oryx/127.6.29.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:44961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.29.190:42115,127.6.29.189:45371
--tserver_master_addrs=127.6.29.190:42115,127.6.29.189:45371,127.6.29.188:44375 with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:55:13.189270 12990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:13.192606 12990 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260925 04:55:13.194734 13205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:13.195242 12990 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20260925 04:55:13.195413 12990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-2/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-2/wal
uuid: "15b0f62dff3a440587c04fb55216d081"
format_stamp: "Formatted at 2026-09-25 04:55:12 on dist-test-slave-xzjr"
server_key: "9600a550ce83c76d9af74cb61a0eba46"
server_key_iv: "f8c5b3a1786a9972618c5539ed904270"
server_key_version: "encryptionkey@0"
I20260925 04:55:13.195904 12990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:13.205308 12990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:55:13.209152 12990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:55:13.242569 12990 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.188:44375
I20260925 04:55:13.242591 13257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.188:44375 every 8 connection(s)
I20260925 04:55:13.243475 12990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/master-2/data/info.pb
I20260925 04:55:13.250734 13258 sys_catalog.cc:263] Verifying existing consensus state
I20260925 04:55:13.256089 13258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081: Bootstrap starting.
I20260925 04:55:13.269795 13258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:55:13.270277 13258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081: Bootstrap complete.
I20260925 04:55:13.271127 13258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: false } } peers { permanent_uuid: "15b0f62dff3a440587c04fb55216d081" member_type: NON_VOTER last_known_addr { host: "127.6.29.188" port: 44375 } attrs { promote: true } }
I20260925 04:55:13.271371 13258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 15b0f62dff3a440587c04fb55216d081, State: Initialized, Role: LEARNER
I20260925 04:55:13.271592 13258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: false } } peers { permanent_uuid: "15b0f62dff3a440587c04fb55216d081" member_type: NON_VOTER last_known_addr { host: "127.6.29.188" port: 44375 } attrs { promote: true } }
I20260925 04:55:13.272378 13261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: false } } peers { permanent_uuid: "15b0f62dff3a440587c04fb55216d081" member_type: NON_VOTER last_known_addr { host: "127.6.29.188" port: 44375 } attrs { promote: true } } }
I20260925 04:55:13.272723 13258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:55:13.272769 13261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [sys.catalog]: This master's current role is: LEARNER
I20260925 04:55:13.281494 13272 catalog_manager.cc:1259] Loaded cluster ID: f48087faeea44abca781b012f0655bbc
I20260925 04:55:13.281612 13272 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081: loading cluster ID for follower catalog manager: success
I20260925 04:55:13.283861 13272 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081: acquiring CA information for follower catalog manager: success
I20260925 04:55:13.285548 13272 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 04:55:13.515956 13160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:13.508371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:44375 (local address 127.0.0.1:57538)
0925 04:55:13.508580 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:13.508601 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:13.508778 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:13.509803 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:13.509809 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:13.509845 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:55:13.510093 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:13.510102 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:13.512157 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:13.512162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:13.513385 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:13.513398 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:13.515133 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:13.515139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:13.515287 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:13.515313 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 04:55:13.515383 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 04:55:13.515485 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 04:55:13.516059 13273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:13.508602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57538 (local address 127.6.29.188:44375)
0925 04:55:13.509349 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:13.509357 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:13.509381 (+ 24us) server_negotiation.cc:407] Connection header received
0925 04:55:13.509467 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:13.509473 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:13.509591 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:13.509799 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:55:13.510254 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:13.511974 (+ 1720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:13.513561 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:13.514955 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:13.515097 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:13.515371 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:13.515434 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 04:55:13.515601 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":394,"threads_started":1}
I20260925 04:55:13.517307 13233 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.7] Dedup: 1.7->[]
I20260925 04:55:13.517990 13261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 932891fca8784f3cbae340352149f1fb. Latest consensus state: current_term: 1 leader_uuid: "932891fca8784f3cbae340352149f1fb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: false } } peers { permanent_uuid: "15b0f62dff3a440587c04fb55216d081" member_type: NON_VOTER last_known_addr { host: "127.6.29.188" port: 44375 } attrs { promote: true } } }
I20260925 04:55:13.518327 13261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [sys.catalog]: This master's current role is: LEARNER
W20260925 04:55:13.609863 13206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:55:13.610206 13206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:55:13.610385 13206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:55:13.618321 13206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:55:13.618508 13206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:55:13.618685 13206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.129
I20260925 04:55:13.628758 13206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:44961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.129.keytab
--principal=oryx/127.6.29.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.129:46555
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.29.129
--webserver_port=33697
--tserver_master_addrs=127.6.29.190:42115,127.6.29.189:45371,127.6.29.188:44375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.129
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:55:13.630383 13206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:55:13.631894 13206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:55:13.645844 13281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:13.645877 13279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:13.645924 13278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:55:13.649463 13206 server_base.cc:1034] running on GCE node
I20260925 04:55:13.650238 13206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:55:13.651295 13206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:55:13.652565 13206 hybrid_clock.cc:648] HybridClock initialized: now 1790312113652492 us; error 72 us; skew 500 ppm
I20260925 04:55:13.652942 13206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:55:13 dist-test-slave-xzjr krb5kdc[12384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312113, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.29.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:55:13.656438 13206 init.cc:377] Logged in from keytab as oryx/127.6.29.129@KRBTEST.COM (short username oryx)
I20260925 04:55:13.658031 13206 webserver.cc:460] Webserver started at http://127.6.29.129:33697/ using document root <none> and password file <none>
I20260925 04:55:13.658828 13206 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:13.658946 13206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:13.664829 13206 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 04:55:13.668075 13288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:13.669514 13206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 04:55:13.669688 13206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-0/wal
uuid: "dad67e0d8c3a4ab8adeda07f8db9a9a6"
format_stamp: "Formatted at 2026-09-25 04:55:10 on dist-test-slave-xzjr"
server_key: "8533d79dea6d8f192b28acb2e9f0afdb"
server_key_iv: "60dfc4667064e7cec8cddc1952b46bcf"
server_key_version: "encryptionkey@0"
I20260925 04:55:13.670701 13206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:13.686318 13206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:55:13.690725 13206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:55:13.691016 13206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:55:13.692075 13206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:55:13.693492 13206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:55:13.693605 13206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:13.693742 13206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:55:13.693835 13206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:13.729195 13206 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.129:46555
I20260925 04:55:13.729288 13400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.129:46555 every 8 connection(s)
I20260925 04:55:13.730777 13206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
I20260925 04:55:13.739092 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 13206
I20260925 04:55:13.739480 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 12624
Sep 25 04:55:13 dist-test-slave-xzjr krb5kdc[12384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312113, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.29.129@KRBTEST.COM for oryx/127.6.29.188@KRBTEST.COM
Sep 25 04:55:13 dist-test-slave-xzjr krb5kdc[12384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312113, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.29.129@KRBTEST.COM for oryx/127.6.29.189@KRBTEST.COM
Sep 25 04:55:13 dist-test-slave-xzjr krb5kdc[12384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312113, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.29.129@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM
Sep 25 04:55:13 dist-test-slave-xzjr krb5kdc[12384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.29.130@KRBTEST.COM
I20260925 04:55:13.775207 13409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:13.734722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:44375 (local address 127.6.29.129:50469)
0925 04:55:13.746716 (+ 11994us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:13.746807 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:13.751813 (+ 5006us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:13.753332 (+ 1519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:13.753350 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:13.754400 (+ 1050us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:13.755530 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:13.755564 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:13.757510 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:13.757514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:13.758948 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:13.758961 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:13.760315 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:13.760322 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:13.760802 (+ 480us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:13.761699 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:13.761746 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:13.765930 (+ 4184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:13.770059 (+ 4129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:13.770072 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:13.770103 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:13.770812 (+ 709us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:13.771307 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:13.771313 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:13.771318 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:13.771621 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:13.772420 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:13.772430 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:13.772924 (+ 494us) client_negotiation.cc:769] Sending connection context
0925 04:55:13.773256 (+ 332us) client_negotiation.cc:240] Negotiation successful
0925 04:55:13.773835 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11505,"thread_start_us":21057,"threads_started":1}
I20260925 04:55:13.776381 13273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:13.735733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:50469 (local address 127.6.29.188:44375)
0925 04:55:13.735984 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:13.735993 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:13.746995 (+ 11002us) server_negotiation.cc:407] Connection header received
0925 04:55:13.752333 (+ 5338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:13.752344 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:13.752472 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:13.752563 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:13.756529 (+ 3966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:13.757308 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:13.759109 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:13.759796 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:13.759937 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:13.766222 (+ 6285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:13.766228 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:13.766234 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:13.766279 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:13.769880 (+ 3601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:13.770978 (+ 1098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:13.770985 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:13.770990 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:13.771164 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:13.771796 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:13.771803 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:13.771808 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:13.772187 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:13.772344 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:13.775729 (+ 3385us) server_negotiation.cc:299] Negotiation successful
0925 04:55:13.775997 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260925 04:55:13.778252 13401 heartbeater.cc:347] Connected to a master server at 127.6.29.188:44375
I20260925 04:55:13.778646 13401 heartbeater.cc:464] Registering TS with master...
I20260925 04:55:13.779402 13406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:13.735007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:43865 (local address 127.6.29.189:45371)
0925 04:55:13.760967 (+ 25960us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:13.760973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:13.760999 (+ 26us) server_negotiation.cc:407] Connection header received
0925 04:55:13.761088 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:13.761094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:13.761185 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:13.762569 (+ 1384us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:13.762779 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:13.763668 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:13.765711 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:13.766319 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:13.766450 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:13.770759 (+ 4309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:13.770767 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:13.770771 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:13.770811 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:13.774734 (+ 3923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:13.777239 (+ 2505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:13.777244 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:13.777249 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:13.777417 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:13.777861 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:13.777867 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:13.777870 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:13.778255 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:13.778459 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:13.778821 (+ 362us) server_negotiation.cc:299] Negotiation successful
0925 04:55:13.779031 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25840,"thread_start_us":25740,"threads_started":1}
I20260925 04:55:13.779547 13407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:13.734722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:45371 (local address 127.6.29.129:43865)
0925 04:55:13.756249 (+ 21527us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:13.756266 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:13.756353 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:13.761357 (+ 5004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:13.761362 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:13.762078 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:13.762397 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:13.762409 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:13.763838 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:13.763843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:13.765539 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:13.765552 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:13.766522 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:13.766528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:13.766709 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:13.767960 (+ 1251us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:13.767990 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:13.770553 (+ 2563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:13.774923 (+ 4370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:13.774929 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:13.774934 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:13.776115 (+ 1181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:13.777565 (+ 1450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:13.777571 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:13.777575 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:13.777715 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:13.778388 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:13.778395 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:13.778562 (+ 167us) client_negotiation.cc:769] Sending connection context
0925 04:55:13.778704 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 04:55:13.779031 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21358,"mutex_wait_us":60,"spinlock_wait_cycles":25344,"thread_start_us":25117,"threads_started":1}
I20260925 04:55:13.779642 13401 heartbeater.cc:511] Master 127.6.29.188:44375 requested a full tablet report, sending...
I20260925 04:55:13.780548 13403 heartbeater.cc:347] Connected to a master server at 127.6.29.189:45371
I20260925 04:55:13.780678 13403 heartbeater.cc:464] Registering TS with master...
I20260925 04:55:13.781136 13403 heartbeater.cc:511] Master 127.6.29.189:45371 requested a full tablet report, sending...
I20260925 04:55:13.782016 13408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:13.735219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.6.29.129:42023)
0925 04:55:13.760354 (+ 25135us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:13.760373 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:13.760546 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:13.763894 (+ 3348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:13.763898 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:13.764519 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:13.764825 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:13.764836 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:13.766997 (+ 2161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:13.767002 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:13.768407 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:13.768426 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:13.769507 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:13.769514 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:13.769687 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:13.770801 (+ 1114us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:13.770822 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:13.773672 (+ 2850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:13.778639 (+ 4967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:13.778645 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:13.778648 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:13.779209 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:13.780109 (+ 900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:13.780114 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:13.780117 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:13.780278 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:13.781043 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:13.781051 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:13.781238 (+ 187us) client_negotiation.cc:769] Sending connection context
0925 04:55:13.781374 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 04:55:13.781645 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25004,"thread_start_us":11825,"threads_started":1}
I20260925 04:55:13.782131 13223 ts_manager.cc:194] Registered new tserver with Master: dad67e0d8c3a4ab8adeda07f8db9a9a6 (127.6.29.129:46555)
I20260925 04:55:13.782848 13034 ts_manager.cc:194] Registered new tserver with Master: dad67e0d8c3a4ab8adeda07f8db9a9a6 (127.6.29.129:46555)
I20260925 04:55:13.783236 13405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:13.733958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:42023 (local address 127.6.29.190:42115)
0925 04:55:13.734401 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:13.734406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:13.761357 (+ 26951us) server_negotiation.cc:407] Connection header received
0925 04:55:13.761438 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:13.761444 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:13.761528 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:13.761646 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:13.765007 (+ 3361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:13.766142 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:13.768604 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:13.769306 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:13.769447 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:13.774439 (+ 4992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:13.774450 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:13.774456 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:13.774502 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:13.778504 (+ 4002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:13.779646 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:13.779652 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:13.779655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:13.779967 (+ 312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:13.780400 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:13.780405 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:13.780408 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:13.780752 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:13.780885 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:13.782775 (+ 1890us) server_negotiation.cc:299] Negotiation successful
0925 04:55:13.782967 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":241,"threads_started":1}
I20260925 04:55:13.783946 13402 heartbeater.cc:347] Connected to a master server at 127.6.29.190:42115
I20260925 04:55:13.784070 13402 heartbeater.cc:464] Registering TS with master...
I20260925 04:55:13.784534 13402 heartbeater.cc:511] Master 127.6.29.190:42115 requested a full tablet report, sending...
I20260925 04:55:13.785513 12429 ts_manager.cc:194] Re-registered known tserver with Master: dad67e0d8c3a4ab8adeda07f8db9a9a6 (127.6.29.129:46555)
I20260925 04:55:13.786103 13411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:13.740617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43176 (local address 127.6.29.130:45893)
0925 04:55:13.759857 (+ 19240us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:13.759869 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:13.760067 (+ 198us) server_negotiation.cc:407] Connection header received
0925 04:55:13.760542 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:13.760556 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:13.760911 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:13.761088 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:13.762890 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:13.764290 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:13.766138 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:13.767060 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:13.767223 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:13.775943 (+ 8720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:13.775962 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:13.775980 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:13.776040 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:13.782371 (+ 6331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:13.783886 (+ 1515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:13.783893 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:13.783895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:13.784053 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:13.784463 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:13.784466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:13.784468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:13.784866 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:13.785166 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:13.785490 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 04:55:13.785769 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19033,"thread_start_us":18857,"threads_started":1}
I20260925 04:55:13.786540 12429 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.29.129@KRBTEST.COM'} at 127.6.29.129:42023
I20260925 04:55:13.787330 12747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:55:13.872174 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 12624
I20260925 04:55:13.889616 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.130:45893
--local_ip_for_outbound_sockets=127.6.29.130
--tserver_master_addrs=127.6.29.190:42115
--webserver_port=40581
--webserver_interface=127.6.29.130
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.130.keytab
--principal=oryx/127.6.29.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:44961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.29.190:42115,127.6.29.189:45371
--tserver_master_addrs=127.6.29.190:42115,127.6.29.189:45371,127.6.29.188:44375 with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:55:14.105870 13414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "932891fca8784f3cbae340352149f1fb" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: false } } peers { permanent_uuid: "15b0f62dff3a440587c04fb55216d081" member_type: NON_VOTER last_known_addr { host: "127.6.29.188" port: 44375 } attrs { promote: true } } }
I20260925 04:55:14.106437 13414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [sys.catalog]: This master's current role is: LEADER
I20260925 04:55:14.107661 13415 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb: attempting to promote NON_VOTER 15b0f62dff3a440587c04fb55216d081 to VOTER
I20260925 04:55:14.108361 13415 consensus_queue.cc:237] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: false } } peers { permanent_uuid: "15b0f62dff3a440587c04fb55216d081" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 44375 } attrs { promote: false } }
I20260925 04:55:14.110669 13233 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [term 1 LEARNER]: Refusing update from remote peer 932891fca8784f3cbae340352149f1fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260925 04:55:14.110965 13044 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [term 1 FOLLOWER]: Refusing update from remote peer 932891fca8784f3cbae340352149f1fb: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260925 04:55:14.111326 13416 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "15b0f62dff3a440587c04fb55216d081" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 44375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 04:55:14.111845 13414 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 04:55:14.116289 13414 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 15b0f62dff3a440587c04fb55216d081 (127.6.29.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: false } } peers { permanent_uuid: "15b0f62dff3a440587c04fb55216d081" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 44375 } attrs { promote: false } } }
I20260925 04:55:14.117137 13044 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 15b0f62dff3a440587c04fb55216d081 (127.6.29.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: false } } peers { permanent_uuid: "15b0f62dff3a440587c04fb55216d081" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 44375 } attrs { promote: false } } }
I20260925 04:55:14.119390 13414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "932891fca8784f3cbae340352149f1fb" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: false } } peers { permanent_uuid: "15b0f62dff3a440587c04fb55216d081" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 44375 } attrs { promote: false } } }
I20260925 04:55:14.119343 13416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "932891fca8784f3cbae340352149f1fb" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: false } } peers { permanent_uuid: "15b0f62dff3a440587c04fb55216d081" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 44375 } attrs { promote: false } } }
I20260925 04:55:14.119596 13414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [sys.catalog]: This master's current role is: LEADER
I20260925 04:55:14.119676 13416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [sys.catalog]: This master's current role is: LEADER
I20260925 04:55:14.119853 13416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "932891fca8784f3cbae340352149f1fb" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: false } } peers { permanent_uuid: "15b0f62dff3a440587c04fb55216d081" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 44375 } attrs { promote: false } } }
I20260925 04:55:14.120003 13416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb [sys.catalog]: This master's current role is: LEADER
I20260925 04:55:14.120370 13233 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 15b0f62dff3a440587c04fb55216d081 (127.6.29.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: false } } peers { permanent_uuid: "15b0f62dff3a440587c04fb55216d081" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 44375 } attrs { promote: false } } }
I20260925 04:55:14.122797 13421 sys_catalog.cc:455] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "932891fca8784f3cbae340352149f1fb" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: false } } peers { permanent_uuid: "15b0f62dff3a440587c04fb55216d081" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 44375 } attrs { promote: false } } }
I20260925 04:55:14.123257 13421 sys_catalog.cc:458] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:55:14.125411 13422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "932891fca8784f3cbae340352149f1fb" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: false } } peers { permanent_uuid: "15b0f62dff3a440587c04fb55216d081" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 44375 } attrs { promote: false } } }
I20260925 04:55:14.125774 13422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 04:55:14.239588 13413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:55:14.240008 13413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:55:14.240123 13413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:55:14.247901 13413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:55:14.248029 13413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:55:14.248237 13413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.130
I20260925 04:55:14.258320 13413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:44961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.130.keytab
--principal=oryx/127.6.29.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.130:45893
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.29.130
--webserver_port=40581
--tserver_master_addrs=127.6.29.190:42115,127.6.29.189:45371,127.6.29.188:44375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.130
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:55:14.259948 13413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:55:14.261325 13413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:55:14.274932 13430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:14.274896 13427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:14.274972 13428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:55:14.276131 13413 server_base.cc:1034] running on GCE node
I20260925 04:55:14.276791 13413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:55:14.277830 13413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:55:14.279027 13413 hybrid_clock.cc:648] HybridClock initialized: now 1790312114278969 us; error 60 us; skew 500 ppm
I20260925 04:55:14.279313 13413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:55:14 dist-test-slave-xzjr krb5kdc[12384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312114, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.29.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:55:14.282518 13413 init.cc:377] Logged in from keytab as oryx/127.6.29.130@KRBTEST.COM (short username oryx)
I20260925 04:55:14.284149 13413 webserver.cc:460] Webserver started at http://127.6.29.130:40581/ using document root <none> and password file <none>
I20260925 04:55:14.284876 13413 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:14.285018 13413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:14.290829 13413 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 04:55:14.293722 13437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:14.295217 13413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:55:14.295409 13413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-1/wal
uuid: "2f7cb902838c4ca382524ae7baa660e4"
format_stamp: "Formatted at 2026-09-25 04:55:11 on dist-test-slave-xzjr"
server_key: "a2eafab9293d8d5722689e6fdec0896a"
server_key_iv: "e190b75485b0b470fc6a7f3e677af2ee"
server_key_version: "encryptionkey@0"
I20260925 04:55:14.296536 13413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:14.364931 13413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:55:14.369421 13413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:55:14.369653 13413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:55:14.370592 13413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:55:14.372071 13413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:55:14.372159 13413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:14.372236 13413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:55:14.372289 13413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:14.407810 13413 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.130:45893
I20260925 04:55:14.407898 13549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.130:45893 every 8 connection(s)
I20260925 04:55:14.409471 13413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
I20260925 04:55:14.412104 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 13413
I20260925 04:55:14.412396 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 12757
Sep 25 04:55:14 dist-test-slave-xzjr krb5kdc[12384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312114, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.29.130@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM
Sep 25 04:55:14 dist-test-slave-xzjr krb5kdc[12384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.29.131@KRBTEST.COM
Sep 25 04:55:14 dist-test-slave-xzjr krb5kdc[12384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312114, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.29.130@KRBTEST.COM for oryx/127.6.29.188@KRBTEST.COM
I20260925 04:55:14.451351 13555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:14.413564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:33861 (local address 127.6.29.190:42115)
0925 04:55:14.417637 (+ 4073us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:14.417646 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:14.426487 (+ 8841us) server_negotiation.cc:407] Connection header received
0925 04:55:14.429083 (+ 2596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:14.429089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:14.429187 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:14.429322 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:14.431938 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:14.432643 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:14.435162 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:14.435633 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:14.435846 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:14.442982 (+ 7136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:14.442988 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:14.442991 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:14.443034 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:14.446963 (+ 3929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:14.448131 (+ 1168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:14.448137 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:14.448141 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:14.448291 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:14.448753 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:14.448761 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:14.448765 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:14.449039 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:14.449227 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:14.450709 (+ 1482us) server_negotiation.cc:299] Negotiation successful
0925 04:55:14.450958 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3904,"thread_start_us":3767,"threads_started":1}
I20260925 04:55:14.452729 13558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:14.416405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.6.29.130:33861)
0925 04:55:14.426290 (+ 9885us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:14.426342 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:14.428707 (+ 2365us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:14.429654 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:14.429670 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:14.430712 (+ 1042us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:14.431640 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:14.431698 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:14.432800 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:14.432807 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:14.435018 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:14.435038 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:14.435904 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:14.435912 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:14.436309 (+ 397us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:14.439218 (+ 2909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:14.439249 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:14.442690 (+ 3441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:14.447247 (+ 4557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:14.447256 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:14.447276 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:14.447972 (+ 696us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:14.448397 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:14.448403 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:14.448412 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:14.448646 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:14.449219 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:14.449230 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:14.450273 (+ 1043us) client_negotiation.cc:769] Sending connection context
0925 04:55:14.450590 (+ 317us) client_negotiation.cc:240] Negotiation successful
0925 04:55:14.451163 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7383,"thread_start_us":7107,"threads_started":1}
Sep 25 04:55:14 dist-test-slave-xzjr krb5kdc[12384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312114, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.29.130@KRBTEST.COM for oryx/127.6.29.189@KRBTEST.COM
I20260925 04:55:14.454329 13551 heartbeater.cc:347] Connected to a master server at 127.6.29.190:42115
I20260925 04:55:14.454697 13556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:14.413925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45100 (local address 127.6.29.131:44161)
0925 04:55:14.428096 (+ 14171us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:14.428109 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:14.428154 (+ 45us) server_negotiation.cc:407] Connection header received
0925 04:55:14.428238 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:14.428251 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:14.428559 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:14.432951 (+ 4392us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:14.433231 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:14.434828 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:14.437271 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:14.438413 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:14.439930 (+ 1517us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:14.444850 (+ 4920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:14.444865 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:14.444885 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:14.444939 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:14.448624 (+ 3685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:14.449876 (+ 1252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:14.449892 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:14.449895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:14.450020 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:14.450398 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:14.450402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:14.450407 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:14.450809 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:14.452789 (+ 1980us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:14.453415 (+ 626us) server_negotiation.cc:299] Negotiation successful
0925 04:55:14.453710 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13955,"thread_start_us":13924,"threads_started":1}
I20260925 04:55:14.454699 13551 heartbeater.cc:464] Registering TS with master...
I20260925 04:55:14.455690 13551 heartbeater.cc:511] Master 127.6.29.190:42115 requested a full tablet report, sending...
I20260925 04:55:14.456302 12880 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:55:14.457636 12429 ts_manager.cc:194] Re-registered known tserver with Master: 2f7cb902838c4ca382524ae7baa660e4 (127.6.29.130:45893)
I20260925 04:55:14.459084 12429 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.29.130@KRBTEST.COM'} at 127.6.29.130:33861
I20260925 04:55:14.462308 13559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:14.417181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:44375 (local address 127.6.29.130:50605)
0925 04:55:14.437799 (+ 20618us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:14.437825 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:14.437985 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:14.439694 (+ 1709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:14.439699 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:14.441643 (+ 1944us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:14.441972 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:14.441982 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:14.443123 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:14.443129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:14.445929 (+ 2800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:14.445945 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:14.447157 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:14.447163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:14.447329 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:14.447978 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:14.448001 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:14.450481 (+ 2480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:14.454642 (+ 4161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:14.454646 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:14.454649 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:14.455225 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:14.455673 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:14.455684 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:14.455688 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:14.455872 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:14.460748 (+ 4876us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:14.460756 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:14.460967 (+ 211us) client_negotiation.cc:769] Sending connection context
0925 04:55:14.461156 (+ 189us) client_negotiation.cc:240] Negotiation successful
0925 04:55:14.461567 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20455,"mutex_wait_us":36,"thread_start_us":20228,"threads_started":1}
I20260925 04:55:14.463454 13561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:14.421517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:48669 (local address 127.6.29.189:45371)
0925 04:55:14.422130 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:14.422139 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:14.438824 (+ 16685us) server_negotiation.cc:407] Connection header received
0925 04:55:14.438908 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:14.438913 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:14.439019 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:14.439152 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:14.443314 (+ 4162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:14.444275 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:14.446301 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:14.446928 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:14.447030 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:14.455332 (+ 8302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:14.455339 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:14.455343 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:14.455385 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:14.459874 (+ 4489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:14.460744 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:14.460750 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:14.460753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:14.460893 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:14.461332 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:14.461336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:14.461339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:14.461692 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:14.462011 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:14.462580 (+ 569us) server_negotiation.cc:299] Negotiation successful
0925 04:55:14.462862 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":288,"threads_started":1}
I20260925 04:55:14.463531 13557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:14.414725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:50605 (local address 127.6.29.188:44375)
0925 04:55:14.439186 (+ 24461us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:14.439194 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:14.439213 (+ 19us) server_negotiation.cc:407] Connection header received
0925 04:55:14.439273 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:14.439277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:14.439358 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:14.439466 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:14.442132 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:14.442971 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:14.446106 (+ 3135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:14.446782 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:14.446902 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:14.450694 (+ 3792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:14.450700 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:14.450704 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:14.450743 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:14.454472 (+ 3729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:14.455363 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:14.455371 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:14.455376 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:14.455524 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:14.459998 (+ 4474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:14.460007 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:14.460013 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:14.460491 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:14.460670 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:14.462770 (+ 2100us) server_negotiation.cc:299] Negotiation successful
0925 04:55:14.463062 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24338,"thread_start_us":24163,"threads_started":1}
I20260925 04:55:14.464448 13560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:14.417986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:45371 (local address 127.6.29.130:48669)
0925 04:55:14.438567 (+ 20581us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:14.438589 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:14.438718 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:14.439387 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:14.439394 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:14.441351 (+ 1957us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:14.441737 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:14.441752 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:14.444418 (+ 2666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:14.444426 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:14.446156 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:14.446167 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:14.448226 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:14.448236 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:14.448855 (+ 619us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:14.451968 (+ 3113us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:14.451989 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:14.455092 (+ 3103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:14.460063 (+ 4971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:14.460067 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:14.460071 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:14.460591 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:14.461041 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:14.461059 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:14.461065 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:14.461205 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:14.462108 (+ 903us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:14.462113 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:14.462259 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 04:55:14.463715 (+ 1456us) client_negotiation.cc:240] Negotiation successful
0925 04:55:14.464063 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20429,"mutex_wait_us":80,"thread_start_us":20230,"threads_started":1}
I20260925 04:55:14.464818 13550 heartbeater.cc:347] Connected to a master server at 127.6.29.188:44375
I20260925 04:55:14.464991 13550 heartbeater.cc:464] Registering TS with master...
I20260925 04:55:14.465289 13552 heartbeater.cc:347] Connected to a master server at 127.6.29.189:45371
I20260925 04:55:14.465404 13552 heartbeater.cc:464] Registering TS with master...
I20260925 04:55:14.465464 13550 heartbeater.cc:511] Master 127.6.29.188:44375 requested a full tablet report, sending...
I20260925 04:55:14.465780 13552 heartbeater.cc:511] Master 127.6.29.189:45371 requested a full tablet report, sending...
I20260925 04:55:14.466508 13223 ts_manager.cc:194] Registered new tserver with Master: 2f7cb902838c4ca382524ae7baa660e4 (127.6.29.130:45893)
I20260925 04:55:14.466866 13034 ts_manager.cc:194] Registered new tserver with Master: 2f7cb902838c4ca382524ae7baa660e4 (127.6.29.130:45893)
I20260925 04:55:14.546191 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 12757
I20260925 04:55:14.562654 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.131:44161
--local_ip_for_outbound_sockets=127.6.29.131
--tserver_master_addrs=127.6.29.190:42115
--webserver_port=37041
--webserver_interface=127.6.29.131
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.131.keytab
--principal=oryx/127.6.29.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:44961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.29.190:42115,127.6.29.189:45371
--tserver_master_addrs=127.6.29.190:42115,127.6.29.189:45371,127.6.29.188:44375 with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:55:14.789115 13402 heartbeater.cc:503] Master 127.6.29.190:42115 was elected leader, sending a full tablet report...
W20260925 04:55:14.908643 13563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:55:14.908953 13563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:55:14.909114 13563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:55:14.916976 13563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:55:14.917098 13563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:55:14.917222 13563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.131
I20260925 04:55:14.927536 13563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:44961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.131.keytab
--principal=oryx/127.6.29.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.131:44161
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.29.131
--webserver_port=37041
--tserver_master_addrs=127.6.29.190:42115,127.6.29.189:45371,127.6.29.188:44375
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.131
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:55:14.928987 13563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:55:14.930289 13563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:55:14.944958 13568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:14.945007 13569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:14.944926 13571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:55:14.946171 13563 server_base.cc:1034] running on GCE node
I20260925 04:55:14.946825 13563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:55:14.947960 13563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:55:14.949198 13563 hybrid_clock.cc:648] HybridClock initialized: now 1790312114949160 us; error 37 us; skew 500 ppm
I20260925 04:55:14.949594 13563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:55:14 dist-test-slave-xzjr krb5kdc[12384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312114, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.29.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:55:14.953130 13563 init.cc:377] Logged in from keytab as oryx/127.6.29.131@KRBTEST.COM (short username oryx)
I20260925 04:55:14.954703 13563 webserver.cc:460] Webserver started at http://127.6.29.131:37041/ using document root <none> and password file <none>
I20260925 04:55:14.955502 13563 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:14.955665 13563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:14.961540 13563 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:55:14.964756 13578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:14.966470 13563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:55:14.966719 13563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-2/wal
uuid: "fbde0dd6885c4cd9ab573d65bfc3231e"
format_stamp: "Formatted at 2026-09-25 04:55:11 on dist-test-slave-xzjr"
server_key: "9d1c0be5fde281a763df21cb7f672c18"
server_key_iv: "c2261f32244882c7bd81c455ed04d834"
server_key_version: "encryptionkey@0"
I20260925 04:55:14.968106 13563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:14.991130 13563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:55:14.995519 13563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:55:14.995837 13563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:55:14.996861 13563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:55:14.998267 13563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:55:14.998351 13563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:14.998423 13563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:55:14.998500 13563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:15.033243 13563 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.131:44161
I20260925 04:55:15.033308 13690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.131:44161 every 8 connection(s)
I20260925 04:55:15.034878 13563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
I20260925 04:55:15.045217 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 13563
Sep 25 04:55:15 dist-test-slave-xzjr krb5kdc[12384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312114, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.29.131@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM
Sep 25 04:55:15 dist-test-slave-xzjr krb5kdc[12384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312114, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.29.131@KRBTEST.COM for oryx/127.6.29.189@KRBTEST.COM
Sep 25 04:55:15 dist-test-slave-xzjr krb5kdc[12384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312114, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.29.131@KRBTEST.COM for oryx/127.6.29.188@KRBTEST.COM
I20260925 04:55:15.094732 13707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:15.038153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:37337 (local address 127.6.29.190:42115)
0925 04:55:15.070085 (+ 31932us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:15.070092 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:15.070131 (+ 39us) server_negotiation.cc:407] Connection header received
0925 04:55:15.070211 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:15.070219 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:15.070311 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:15.070417 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:15.075552 (+ 5135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.076929 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.079576 (+ 2647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.080256 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.080376 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:15.085629 (+ 5253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:15.085636 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:15.085642 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:15.085685 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:15.090302 (+ 4617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:15.091460 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:15.091465 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:15.091469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:15.091619 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:15.092259 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:15.092266 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:15.092270 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:15.092905 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:15.093022 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:15.094092 (+ 1070us) server_negotiation.cc:299] Negotiation successful
0925 04:55:15.094344 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31784,"thread_start_us":35577,"threads_started":1}
I20260925 04:55:15.097640 13698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:15.038672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.6.29.131:37337)
0925 04:55:15.054544 (+ 15872us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:15.054633 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:15.057706 (+ 3073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:15.073919 (+ 16213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:15.073924 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:15.074689 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:15.075345 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:15.075372 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.077137 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.077142 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:15.079414 (+ 2272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:15.079427 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.080728 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.080737 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:15.081057 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:15.081903 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:15.081937 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:15.085352 (+ 3415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:15.090504 (+ 5152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:15.090515 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:15.090540 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:15.091315 (+ 775us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:15.091807 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:15.091812 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:15.091815 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:15.092116 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:15.093144 (+ 1028us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:15.093152 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:15.093640 (+ 488us) client_negotiation.cc:769] Sending connection context
0925 04:55:15.095262 (+ 1622us) client_negotiation.cc:240] Negotiation successful
0925 04:55:15.095922 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15428,"mutex_wait_us":68,"thread_start_us":15206,"threads_started":1}
I20260925 04:55:15.097745 13697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:15.038871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:60111 (local address 127.6.29.189:45371)
0925 04:55:15.050334 (+ 11463us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:15.050344 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:15.075177 (+ 24833us) server_negotiation.cc:407] Connection header received
0925 04:55:15.075485 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:15.075492 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:15.075575 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:15.075674 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:15.077015 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.078045 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.081823 (+ 3778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.082507 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.082664 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:15.089290 (+ 6626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:15.089297 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:15.089301 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:15.089342 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:15.094452 (+ 5110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:15.095285 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:15.095290 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:15.095293 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:15.095448 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:15.095952 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:15.095958 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:15.095961 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:15.096381 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:15.096512 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:15.097027 (+ 515us) server_negotiation.cc:299] Negotiation successful
0925 04:55:15.097321 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11285,"thread_start_us":11166,"threads_started":1}
I20260925 04:55:15.100347 13692 heartbeater.cc:347] Connected to a master server at 127.6.29.190:42115
I20260925 04:55:15.101085 13695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:15.037987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:37097 (local address 127.6.29.188:44375)
0925 04:55:15.053777 (+ 15790us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:15.053787 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:15.070438 (+ 16651us) server_negotiation.cc:407] Connection header received
0925 04:55:15.070509 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:15.070514 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:15.070622 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:15.070753 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:15.082867 (+ 12114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.084383 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.087200 (+ 2817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.087923 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.088054 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:15.092178 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:15.092186 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:15.092191 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:15.092234 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:15.095979 (+ 3745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:15.096844 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:15.096849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:15.096853 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:15.097002 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:15.097412 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:15.097417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:15.097420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:15.097829 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:15.097957 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:15.099646 (+ 1689us) server_negotiation.cc:299] Negotiation successful
0925 04:55:15.100718 (+ 1072us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15635,"thread_start_us":20172,"threads_started":1}
I20260925 04:55:15.101553 13700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:15.039015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:45371 (local address 127.6.29.131:60111)
0925 04:55:15.075025 (+ 36010us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:15.075046 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:15.075308 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:15.075936 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:15.075940 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:15.076562 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:15.076836 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:15.076847 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.078246 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.078251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:15.080136 (+ 1885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:15.080149 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.082785 (+ 2636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.082795 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:15.083072 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:15.083926 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:15.083950 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:15.089043 (+ 5093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:15.094623 (+ 5580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:15.094631 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:15.094636 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:15.095142 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:15.095606 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:15.095620 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:15.095624 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:15.095821 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:15.096593 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:15.096599 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:15.096768 (+ 169us) client_negotiation.cc:769] Sending connection context
0925 04:55:15.098007 (+ 1239us) client_negotiation.cc:240] Negotiation successful
0925 04:55:15.099326 (+ 1319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35275,"mutex_wait_us":345,"thread_start_us":34094,"threads_started":1}
I20260925 04:55:15.101735 13692 heartbeater.cc:464] Registering TS with master...
I20260925 04:55:15.103030 13693 heartbeater.cc:347] Connected to a master server at 127.6.29.189:45371
I20260925 04:55:15.103122 13692 heartbeater.cc:511] Master 127.6.29.190:42115 requested a full tablet report, sending...
I20260925 04:55:15.103176 13693 heartbeater.cc:464] Registering TS with master...
I20260925 04:55:15.103685 13693 heartbeater.cc:511] Master 127.6.29.189:45371 requested a full tablet report, sending...
I20260925 04:55:15.102015 13699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:15.038672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:44375 (local address 127.6.29.131:37097)
0925 04:55:15.070133 (+ 31461us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:15.070157 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:15.070309 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:15.071744 (+ 1435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:15.071783 (+ 39us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:15.074122 (+ 2339us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:15.078919 (+ 4797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:15.078932 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.084546 (+ 5614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.084551 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:15.086992 (+ 2441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:15.087009 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.088325 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.088331 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:15.088514 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:15.089098 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:15.089124 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:15.091951 (+ 2827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:15.096204 (+ 4253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:15.096208 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:15.096211 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:15.096714 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:15.097122 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:15.097127 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:15.097130 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:15.097289 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:15.098026 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:15.098030 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:15.098175 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 04:55:15.098759 (+ 584us) client_negotiation.cc:240] Negotiation successful
0925 04:55:15.099947 (+ 1188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31300,"mutex_wait_us":976,"spinlock_wait_cycles":28544,"thread_start_us":32102,"threads_started":1}
I20260925 04:55:15.105059 12429 ts_manager.cc:194] Re-registered known tserver with Master: fbde0dd6885c4cd9ab573d65bfc3231e (127.6.29.131:44161)
I20260925 04:55:15.105131 13691 heartbeater.cc:347] Connected to a master server at 127.6.29.188:44375
I20260925 04:55:15.105206 13034 ts_manager.cc:194] Registered new tserver with Master: fbde0dd6885c4cd9ab573d65bfc3231e (127.6.29.131:44161)
I20260925 04:55:15.105274 13691 heartbeater.cc:464] Registering TS with master...
I20260925 04:55:15.105682 13691 heartbeater.cc:511] Master 127.6.29.188:44375 requested a full tablet report, sending...
I20260925 04:55:15.106354 12429 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.29.131@KRBTEST.COM'} at 127.6.29.131:37337
I20260925 04:55:15.106891 13223 ts_manager.cc:194] Registered new tserver with Master: fbde0dd6885c4cd9ab573d65bfc3231e (127.6.29.131:44161)
I20260925 04:55:15.112494 13696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:15.058893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46564 (local address 127.6.29.190:42115)
0925 04:55:15.073717 (+ 14824us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:15.073724 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:15.083533 (+ 9809us) server_negotiation.cc:407] Connection header received
0925 04:55:15.083604 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:15.083609 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:15.083695 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:15.083847 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:15.088080 (+ 4233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.089444 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.098688 (+ 9244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.099340 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.099436 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:15.104299 (+ 4863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:15.104308 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:15.104313 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:15.104367 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:15.108330 (+ 3963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:15.109876 (+ 1546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:15.109884 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:15.109888 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:15.110056 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:15.110631 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:15.110636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:15.110640 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:15.110993 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:15.111246 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:15.111675 (+ 429us) server_negotiation.cc:299] Negotiation successful
0925 04:55:15.112072 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14675,"thread_start_us":11263,"threads_started":1}
I20260925 04:55:15.115124 13708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:15.059227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57546 (local address 127.6.29.188:44375)
0925 04:55:15.073717 (+ 14490us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:15.073724 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:15.081928 (+ 8204us) server_negotiation.cc:407] Connection header received
0925 04:55:15.085200 (+ 3272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:15.085205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:15.085286 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:15.085397 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:15.090180 (+ 4783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.091426 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.098340 (+ 6914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.099014 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.099125 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:15.108069 (+ 8944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:15.108076 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:15.108081 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:15.108121 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:15.111924 (+ 3803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:15.113051 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:15.113057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:15.113061 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:15.113204 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:15.113597 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:15.113602 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:15.113613 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:15.113926 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:15.114183 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:15.114532 (+ 349us) server_negotiation.cc:299] Negotiation successful
0925 04:55:15.114752 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14354,"thread_start_us":14486,"threads_started":1}
I20260925 04:55:15.115573 13709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:15.064132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48870 (local address 127.6.29.189:45371)
0925 04:55:15.080641 (+ 16509us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:15.080650 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:15.085726 (+ 5076us) server_negotiation.cc:407] Connection header received
0925 04:55:15.085789 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:15.085796 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:15.085874 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:15.085977 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:15.092011 (+ 6034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.093328 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.098523 (+ 5195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.099844 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.099945 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:15.108345 (+ 8400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:15.108351 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:15.108357 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:15.108403 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:15.111961 (+ 3558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:15.113057 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:15.113063 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:15.113066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:15.113243 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:15.113689 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:15.113694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:15.113700 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:15.114043 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:15.114270 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:15.114866 (+ 596us) server_negotiation.cc:299] Negotiation successful
0925 04:55:15.115121 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16360,"thread_start_us":16221,"threads_started":1}
I20260925 04:55:15.117216 12429 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:55:15.124665 12429 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:55:15.170063 13724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:15.143247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:45893 (local address 127.0.0.1:43178)
0925 04:55:15.161230 (+ 17983us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:15.161250 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:15.161419 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:15.162519 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:15.162524 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:15.162552 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:55:15.163003 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:15.163019 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.164940 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.164946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:15.166877 (+ 1931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:15.166889 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.168695 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.168702 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:15.168893 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:15.168929 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 04:55:15.169031 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 04:55:15.169207 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17840,"thread_start_us":25870,"threads_started":1}
I20260925 04:55:15.170740 13722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:15.143459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43178 (local address 127.6.29.130:45893)
0925 04:55:15.158980 (+ 15521us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:15.158995 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:15.161564 (+ 2569us) server_negotiation.cc:407] Connection header received
0925 04:55:15.161684 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:15.161693 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:15.162010 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:15.162247 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:55:15.163218 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.164758 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.167027 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.168138 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.168258 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:15.168516 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:15.170294 (+ 1778us) server_negotiation.cc:299] Negotiation successful
0925 04:55:15.170477 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15262,"thread_start_us":15551,"threads_started":1}
I20260925 04:55:15.174728 13485 tablet_service.cc:1467] Processing CreateTablet for tablet b1e28619bee34617b532d8f73e950454 (DEFAULT_TABLE table=test-table [id=54d8ce3b175c4205ad4f83a6ce8afce9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:55:15.176453 13485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1e28619bee34617b532d8f73e950454. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:55:15.178157 13723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:15.143800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:46555 (local address 127.0.0.1:54682)
0925 04:55:15.168755 (+ 24955us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:15.168775 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:15.168895 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:15.169629 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:15.169634 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:15.169661 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:55:15.170078 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:15.170095 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.172559 (+ 2464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.172564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:15.175150 (+ 2586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:15.175168 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.177246 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.177253 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:15.177430 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:15.177465 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 04:55:15.177542 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 04:55:15.177704 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24800,"mutex_wait_us":25,"thread_start_us":24617,"threads_started":1}
I20260925 04:55:15.178447 13721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:15.144194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.131:44161 (local address 127.0.0.1:45116)
0925 04:55:15.169399 (+ 25205us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:15.169416 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:15.169529 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:15.170440 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:15.170446 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:15.170475 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:55:15.170842 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:15.170857 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.172978 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.172983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:15.175272 (+ 2289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:15.175286 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.177572 (+ 2286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.177577 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:15.177752 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:15.177782 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 04:55:15.177878 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 04:55:15.178017 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25070,"mutex_wait_us":31,"thread_start_us":16727,"threads_started":1}
I20260925 04:55:15.178462 13726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:15.144634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45116 (local address 127.6.29.131:44161)
0925 04:55:15.160684 (+ 16050us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:15.160697 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:15.169559 (+ 8862us) server_negotiation.cc:407] Connection header received
0925 04:55:15.169667 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:15.169675 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:15.169959 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:15.170172 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:55:15.171201 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.172805 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.175471 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.177025 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.177150 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:15.177369 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:15.177957 (+ 588us) server_negotiation.cc:299] Negotiation successful
0925 04:55:15.178155 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15827,"thread_start_us":15649,"threads_started":1}
I20260925 04:55:15.179171 13725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:15.144415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54682 (local address 127.6.29.129:46555)
0925 04:55:15.168272 (+ 23857us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:15.168281 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:15.168898 (+ 617us) server_negotiation.cc:407] Connection header received
0925 04:55:15.169072 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:15.169085 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:15.169373 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:15.170592 (+ 1219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:55:15.170810 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.172406 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.175378 (+ 2972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.177018 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.178563 (+ 1545us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:15.178784 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:15.178851 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 04:55:15.178964 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23638,"thread_start_us":23530,"threads_started":1}
I20260925 04:55:15.182677 13336 tablet_service.cc:1467] Processing CreateTablet for tablet b1e28619bee34617b532d8f73e950454 (DEFAULT_TABLE table=test-table [id=54d8ce3b175c4205ad4f83a6ce8afce9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:55:15.183972 13336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1e28619bee34617b532d8f73e950454. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:55:15.186594 13626 tablet_service.cc:1467] Processing CreateTablet for tablet b1e28619bee34617b532d8f73e950454 (DEFAULT_TABLE table=test-table [id=54d8ce3b175c4205ad4f83a6ce8afce9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:55:15.188272 13626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1e28619bee34617b532d8f73e950454. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:55:15.189352 13727 tablet_bootstrap.cc:492] T b1e28619bee34617b532d8f73e950454 P 2f7cb902838c4ca382524ae7baa660e4: Bootstrap starting.
I20260925 04:55:15.192607 13727 tablet_bootstrap.cc:654] T b1e28619bee34617b532d8f73e950454 P 2f7cb902838c4ca382524ae7baa660e4: Neither blocks nor log segments found. Creating new log.
I20260925 04:55:15.193629 13727 log.cc:824] T b1e28619bee34617b532d8f73e950454 P 2f7cb902838c4ca382524ae7baa660e4: Log is configured to *not* fsync() on all Append() calls
I20260925 04:55:15.196230 13727 tablet_bootstrap.cc:492] T b1e28619bee34617b532d8f73e950454 P 2f7cb902838c4ca382524ae7baa660e4: No bootstrap required, opened a new log
I20260925 04:55:15.196560 13727 ts_tablet_manager.cc:1397] T b1e28619bee34617b532d8f73e950454 P 2f7cb902838c4ca382524ae7baa660e4: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20260925 04:55:15.200024 13729 tablet_bootstrap.cc:492] T b1e28619bee34617b532d8f73e950454 P dad67e0d8c3a4ab8adeda07f8db9a9a6: Bootstrap starting.
I20260925 04:55:15.203801 13729 tablet_bootstrap.cc:654] T b1e28619bee34617b532d8f73e950454 P dad67e0d8c3a4ab8adeda07f8db9a9a6: Neither blocks nor log segments found. Creating new log.
I20260925 04:55:15.204279 13730 tablet_bootstrap.cc:492] T b1e28619bee34617b532d8f73e950454 P fbde0dd6885c4cd9ab573d65bfc3231e: Bootstrap starting.
I20260925 04:55:15.205348 13729 log.cc:824] T b1e28619bee34617b532d8f73e950454 P dad67e0d8c3a4ab8adeda07f8db9a9a6: Log is configured to *not* fsync() on all Append() calls
I20260925 04:55:15.207077 13727 raft_consensus.cc:348] T b1e28619bee34617b532d8f73e950454 P 2f7cb902838c4ca382524ae7baa660e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7cb902838c4ca382524ae7baa660e4" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45893 } } peers { permanent_uuid: "dad67e0d8c3a4ab8adeda07f8db9a9a6" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 46555 } } peers { permanent_uuid: "fbde0dd6885c4cd9ab573d65bfc3231e" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 44161 } }
I20260925 04:55:15.207515 13730 tablet_bootstrap.cc:654] T b1e28619bee34617b532d8f73e950454 P fbde0dd6885c4cd9ab573d65bfc3231e: Neither blocks nor log segments found. Creating new log.
I20260925 04:55:15.207531 13727 raft_consensus.cc:374] T b1e28619bee34617b532d8f73e950454 P 2f7cb902838c4ca382524ae7baa660e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:55:15.207625 13727 raft_consensus.cc:729] T b1e28619bee34617b532d8f73e950454 P 2f7cb902838c4ca382524ae7baa660e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f7cb902838c4ca382524ae7baa660e4, State: Initialized, Role: FOLLOWER
I20260925 04:55:15.208489 13727 consensus_queue.cc:260] T b1e28619bee34617b532d8f73e950454 P 2f7cb902838c4ca382524ae7baa660e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7cb902838c4ca382524ae7baa660e4" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45893 } } peers { permanent_uuid: "dad67e0d8c3a4ab8adeda07f8db9a9a6" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 46555 } } peers { permanent_uuid: "fbde0dd6885c4cd9ab573d65bfc3231e" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 44161 } }
I20260925 04:55:15.208851 13730 log.cc:824] T b1e28619bee34617b532d8f73e950454 P fbde0dd6885c4cd9ab573d65bfc3231e: Log is configured to *not* fsync() on all Append() calls
I20260925 04:55:15.209336 13729 tablet_bootstrap.cc:492] T b1e28619bee34617b532d8f73e950454 P dad67e0d8c3a4ab8adeda07f8db9a9a6: No bootstrap required, opened a new log
I20260925 04:55:15.209717 13729 ts_tablet_manager.cc:1397] T b1e28619bee34617b532d8f73e950454 P dad67e0d8c3a4ab8adeda07f8db9a9a6: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260925 04:55:15.212670 13551 heartbeater.cc:503] Master 127.6.29.190:42115 was elected leader, sending a full tablet report...
I20260925 04:55:15.217954 13727 ts_tablet_manager.cc:1428] T b1e28619bee34617b532d8f73e950454 P 2f7cb902838c4ca382524ae7baa660e4: Time spent starting tablet: real 0.021s user 0.012s sys 0.003s
I20260925 04:55:15.218072 13730 tablet_bootstrap.cc:492] T b1e28619bee34617b532d8f73e950454 P fbde0dd6885c4cd9ab573d65bfc3231e: No bootstrap required, opened a new log
I20260925 04:55:15.218348 13730 ts_tablet_manager.cc:1397] T b1e28619bee34617b532d8f73e950454 P fbde0dd6885c4cd9ab573d65bfc3231e: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20260925 04:55:15.220623 13729 raft_consensus.cc:348] T b1e28619bee34617b532d8f73e950454 P dad67e0d8c3a4ab8adeda07f8db9a9a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7cb902838c4ca382524ae7baa660e4" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45893 } } peers { permanent_uuid: "dad67e0d8c3a4ab8adeda07f8db9a9a6" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 46555 } } peers { permanent_uuid: "fbde0dd6885c4cd9ab573d65bfc3231e" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 44161 } }
I20260925 04:55:15.221086 13729 raft_consensus.cc:374] T b1e28619bee34617b532d8f73e950454 P dad67e0d8c3a4ab8adeda07f8db9a9a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:55:15.221222 13729 raft_consensus.cc:729] T b1e28619bee34617b532d8f73e950454 P dad67e0d8c3a4ab8adeda07f8db9a9a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dad67e0d8c3a4ab8adeda07f8db9a9a6, State: Initialized, Role: FOLLOWER
I20260925 04:55:15.221941 13729 consensus_queue.cc:260] T b1e28619bee34617b532d8f73e950454 P dad67e0d8c3a4ab8adeda07f8db9a9a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7cb902838c4ca382524ae7baa660e4" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45893 } } peers { permanent_uuid: "dad67e0d8c3a4ab8adeda07f8db9a9a6" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 46555 } } peers { permanent_uuid: "fbde0dd6885c4cd9ab573d65bfc3231e" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 44161 } }
I20260925 04:55:15.223852 13729 ts_tablet_manager.cc:1428] T b1e28619bee34617b532d8f73e950454 P dad67e0d8c3a4ab8adeda07f8db9a9a6: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260925 04:55:15.228719 13730 raft_consensus.cc:348] T b1e28619bee34617b532d8f73e950454 P fbde0dd6885c4cd9ab573d65bfc3231e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7cb902838c4ca382524ae7baa660e4" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45893 } } peers { permanent_uuid: "dad67e0d8c3a4ab8adeda07f8db9a9a6" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 46555 } } peers { permanent_uuid: "fbde0dd6885c4cd9ab573d65bfc3231e" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 44161 } }
I20260925 04:55:15.229158 13730 raft_consensus.cc:374] T b1e28619bee34617b532d8f73e950454 P fbde0dd6885c4cd9ab573d65bfc3231e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:55:15.229270 13730 raft_consensus.cc:729] T b1e28619bee34617b532d8f73e950454 P fbde0dd6885c4cd9ab573d65bfc3231e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbde0dd6885c4cd9ab573d65bfc3231e, State: Initialized, Role: FOLLOWER
I20260925 04:55:15.229863 13730 consensus_queue.cc:260] T b1e28619bee34617b532d8f73e950454 P fbde0dd6885c4cd9ab573d65bfc3231e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7cb902838c4ca382524ae7baa660e4" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45893 } } peers { permanent_uuid: "dad67e0d8c3a4ab8adeda07f8db9a9a6" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 46555 } } peers { permanent_uuid: "fbde0dd6885c4cd9ab573d65bfc3231e" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 44161 } }
I20260925 04:55:15.230945 13692 heartbeater.cc:503] Master 127.6.29.190:42115 was elected leader, sending a full tablet report...
I20260925 04:55:15.231586 13730 ts_tablet_manager.cc:1428] T b1e28619bee34617b532d8f73e950454 P fbde0dd6885c4cd9ab573d65bfc3231e: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260925 04:55:15.234899 13404 tablet.cc:2369] T b1e28619bee34617b532d8f73e950454 P dad67e0d8c3a4ab8adeda07f8db9a9a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:55:15.287564 13694 tablet.cc:2369] T b1e28619bee34617b532d8f73e950454 P fbde0dd6885c4cd9ab573d65bfc3231e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:55:15.432226 13553 tablet.cc:2369] T b1e28619bee34617b532d8f73e950454 P 2f7cb902838c4ca382524ae7baa660e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:55:15.554025 13735 raft_consensus.cc:482] T b1e28619bee34617b532d8f73e950454 P fbde0dd6885c4cd9ab573d65bfc3231e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:55:15.554384 13735 raft_consensus.cc:504] T b1e28619bee34617b532d8f73e950454 P fbde0dd6885c4cd9ab573d65bfc3231e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7cb902838c4ca382524ae7baa660e4" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45893 } } peers { permanent_uuid: "dad67e0d8c3a4ab8adeda07f8db9a9a6" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 46555 } } peers { permanent_uuid: "fbde0dd6885c4cd9ab573d65bfc3231e" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 44161 } }
I20260925 04:55:15.556154 13735 leader_election.cc:288] T b1e28619bee34617b532d8f73e950454 P fbde0dd6885c4cd9ab573d65bfc3231e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f7cb902838c4ca382524ae7baa660e4 (127.6.29.130:45893), dad67e0d8c3a4ab8adeda07f8db9a9a6 (127.6.29.129:46555)
I20260925 04:55:15.562153 13722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:15.556679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:46173 (local address 127.6.29.130:45893)
0925 04:55:15.556868 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:15.556874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:15.556896 (+ 22us) server_negotiation.cc:407] Connection header received
0925 04:55:15.557013 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:15.557017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:15.557075 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:15.557190 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:55:15.557607 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.558301 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.560283 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.561096 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.561261 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:15.561313 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:15.561739 (+ 426us) server_negotiation.cc:299] Negotiation successful
0925 04:55:15.561894 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 04:55:15.562251 13699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:15.556417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:46555 (local address 127.6.29.131:55851)
0925 04:55:15.556679 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:15.556700 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:15.556829 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:15.557212 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:15.557216 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:15.557253 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:55:15.557458 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:15.557466 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.558457 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.558462 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:15.560117 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:15.560129 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.561439 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.561448 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:15.561737 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:15.561769 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 04:55:15.561852 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 04:55:15.561972 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":81,"spinlock_wait_cycles":1792}
I20260925 04:55:15.562405 13700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:15.556421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:45893 (local address 127.6.29.131:46173)
0925 04:55:15.556755 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:15.556775 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:15.556884 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:15.557212 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:15.557216 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:15.557277 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:55:15.557463 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:15.557470 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.558467 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.558472 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:15.560116 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:15.560133 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.561396 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.561401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:15.561562 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:15.561587 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 04:55:15.561699 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 04:55:15.561869 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":130,"spinlock_wait_cycles":61696}
I20260925 04:55:15.562826 13725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:15.556730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:55851 (local address 127.6.29.129:46555)
0925 04:55:15.556925 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:15.556932 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:15.556959 (+ 27us) server_negotiation.cc:407] Connection header received
0925 04:55:15.557014 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:15.557018 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:15.557072 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:15.557183 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:55:15.557603 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.558305 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.560268 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.561042 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.561228 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:15.561300 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:15.562453 (+ 1153us) server_negotiation.cc:299] Negotiation successful
0925 04:55:15.562598 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 04:55:15.563354 13505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e28619bee34617b532d8f73e950454" candidate_uuid: "fbde0dd6885c4cd9ab573d65bfc3231e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f7cb902838c4ca382524ae7baa660e4" is_pre_election: true
I20260925 04:55:15.563742 13356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e28619bee34617b532d8f73e950454" candidate_uuid: "fbde0dd6885c4cd9ab573d65bfc3231e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dad67e0d8c3a4ab8adeda07f8db9a9a6" is_pre_election: true
I20260925 04:55:15.564049 13505 raft_consensus.cc:2454] T b1e28619bee34617b532d8f73e950454 P 2f7cb902838c4ca382524ae7baa660e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbde0dd6885c4cd9ab573d65bfc3231e in term 0.
I20260925 04:55:15.564354 13356 raft_consensus.cc:2454] T b1e28619bee34617b532d8f73e950454 P dad67e0d8c3a4ab8adeda07f8db9a9a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbde0dd6885c4cd9ab573d65bfc3231e in term 0.
I20260925 04:55:15.564857 13581 leader_election.cc:302] T b1e28619bee34617b532d8f73e950454 P fbde0dd6885c4cd9ab573d65bfc3231e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f7cb902838c4ca382524ae7baa660e4, fbde0dd6885c4cd9ab573d65bfc3231e; no voters:
I20260925 04:55:15.565313 13735 raft_consensus.cc:2781] T b1e28619bee34617b532d8f73e950454 P fbde0dd6885c4cd9ab573d65bfc3231e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:55:15.565493 13735 raft_consensus.cc:482] T b1e28619bee34617b532d8f73e950454 P fbde0dd6885c4cd9ab573d65bfc3231e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:55:15.565608 13735 raft_consensus.cc:3037] T b1e28619bee34617b532d8f73e950454 P fbde0dd6885c4cd9ab573d65bfc3231e [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:55:15.567948 13735 raft_consensus.cc:504] T b1e28619bee34617b532d8f73e950454 P fbde0dd6885c4cd9ab573d65bfc3231e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7cb902838c4ca382524ae7baa660e4" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45893 } } peers { permanent_uuid: "dad67e0d8c3a4ab8adeda07f8db9a9a6" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 46555 } } peers { permanent_uuid: "fbde0dd6885c4cd9ab573d65bfc3231e" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 44161 } }
I20260925 04:55:15.568610 13735 leader_election.cc:288] T b1e28619bee34617b532d8f73e950454 P fbde0dd6885c4cd9ab573d65bfc3231e [CANDIDATE]: Term 1 election: Requested vote from peers 2f7cb902838c4ca382524ae7baa660e4 (127.6.29.130:45893), dad67e0d8c3a4ab8adeda07f8db9a9a6 (127.6.29.129:46555)
I20260925 04:55:15.569141 13356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e28619bee34617b532d8f73e950454" candidate_uuid: "fbde0dd6885c4cd9ab573d65bfc3231e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dad67e0d8c3a4ab8adeda07f8db9a9a6"
I20260925 04:55:15.569167 13505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e28619bee34617b532d8f73e950454" candidate_uuid: "fbde0dd6885c4cd9ab573d65bfc3231e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f7cb902838c4ca382524ae7baa660e4"
I20260925 04:55:15.569350 13356 raft_consensus.cc:3037] T b1e28619bee34617b532d8f73e950454 P dad67e0d8c3a4ab8adeda07f8db9a9a6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:55:15.569496 13505 raft_consensus.cc:3037] T b1e28619bee34617b532d8f73e950454 P 2f7cb902838c4ca382524ae7baa660e4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:55:15.571739 13505 raft_consensus.cc:2454] T b1e28619bee34617b532d8f73e950454 P 2f7cb902838c4ca382524ae7baa660e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbde0dd6885c4cd9ab573d65bfc3231e in term 1.
I20260925 04:55:15.571745 13356 raft_consensus.cc:2454] T b1e28619bee34617b532d8f73e950454 P dad67e0d8c3a4ab8adeda07f8db9a9a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbde0dd6885c4cd9ab573d65bfc3231e in term 1.
I20260925 04:55:15.572301 13581 leader_election.cc:302] T b1e28619bee34617b532d8f73e950454 P fbde0dd6885c4cd9ab573d65bfc3231e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f7cb902838c4ca382524ae7baa660e4, fbde0dd6885c4cd9ab573d65bfc3231e; no voters:
I20260925 04:55:15.572685 13735 raft_consensus.cc:2781] T b1e28619bee34617b532d8f73e950454 P fbde0dd6885c4cd9ab573d65bfc3231e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:55:15.573153 13735 raft_consensus.cc:686] T b1e28619bee34617b532d8f73e950454 P fbde0dd6885c4cd9ab573d65bfc3231e [term 1 LEADER]: Becoming Leader. State: Replica: fbde0dd6885c4cd9ab573d65bfc3231e, State: Running, Role: LEADER
I20260925 04:55:15.573798 13735 consensus_queue.cc:237] T b1e28619bee34617b532d8f73e950454 P fbde0dd6885c4cd9ab573d65bfc3231e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7cb902838c4ca382524ae7baa660e4" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45893 } } peers { permanent_uuid: "dad67e0d8c3a4ab8adeda07f8db9a9a6" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 46555 } } peers { permanent_uuid: "fbde0dd6885c4cd9ab573d65bfc3231e" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 44161 } }
I20260925 04:55:15.589901 12429 catalog_manager.cc:5495] T b1e28619bee34617b532d8f73e950454 P fbde0dd6885c4cd9ab573d65bfc3231e reported cstate change: term changed from 0 to 1, leader changed from <none> to fbde0dd6885c4cd9ab573d65bfc3231e (127.6.29.131). New cstate: current_term: 1 leader_uuid: "fbde0dd6885c4cd9ab573d65bfc3231e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7cb902838c4ca382524ae7baa660e4" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dad67e0d8c3a4ab8adeda07f8db9a9a6" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 46555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbde0dd6885c4cd9ab573d65bfc3231e" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 44161 } health_report { overall_health: HEALTHY } } }
I20260925 04:55:15.624898 13726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:15.618858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45118 (local address 127.6.29.131:44161)
0925 04:55:15.619020 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:15.619025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:15.619490 (+ 465us) server_negotiation.cc:407] Connection header received
0925 04:55:15.619713 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:15.619717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:15.619876 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:15.620003 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:55:15.620379 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.621093 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.622447 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.622854 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.622944 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:15.623469 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:55:15.624374 (+ 905us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:55:15.624513 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:15.624605 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 04:55:15.624676 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 04:55:15.633476 13356 raft_consensus.cc:1261] T b1e28619bee34617b532d8f73e950454 P dad67e0d8c3a4ab8adeda07f8db9a9a6 [term 1 FOLLOWER]: Refusing update from remote peer fbde0dd6885c4cd9ab573d65bfc3231e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:55:15.633476 13505 raft_consensus.cc:1261] T b1e28619bee34617b532d8f73e950454 P 2f7cb902838c4ca382524ae7baa660e4 [term 1 FOLLOWER]: Refusing update from remote peer fbde0dd6885c4cd9ab573d65bfc3231e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:55:15.634524 13736 consensus_queue.cc:1035] T b1e28619bee34617b532d8f73e950454 P fbde0dd6885c4cd9ab573d65bfc3231e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f7cb902838c4ca382524ae7baa660e4" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:55:15.635008 13735 consensus_queue.cc:1035] T b1e28619bee34617b532d8f73e950454 P fbde0dd6885c4cd9ab573d65bfc3231e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dad67e0d8c3a4ab8adeda07f8db9a9a6" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 46555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:55:15.651309 13742 mvcc.cc:205] Tried to move back new op lower bound from 7333118425623216128 to 7333118425394487296. Current Snapshot: MvccSnapshot[applied={T|T < 7333118425623216128}]
I20260925 04:55:15.668597 12429 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:55:15.668888 12429 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:55:15.676155 12429 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 932891fca8784f3cbae340352149f1fb: Sending DeleteTablet for 3 replicas of tablet b1e28619bee34617b532d8f73e950454
I20260925 04:55:15.677325 13336 tablet_service.cc:1514] Processing DeleteTablet for tablet b1e28619bee34617b532d8f73e950454 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:55:15 UTC) from {username='oryx', principal='oryx/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:54682
I20260925 04:55:15.677280 13485 tablet_service.cc:1514] Processing DeleteTablet for tablet b1e28619bee34617b532d8f73e950454 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:55:15 UTC) from {username='oryx', principal='oryx/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:43178
I20260925 04:55:15.677374 13626 tablet_service.cc:1514] Processing DeleteTablet for tablet b1e28619bee34617b532d8f73e950454 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:55:15 UTC) from {username='oryx', principal='oryx/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:45116
I20260925 04:55:15.708029 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 12397
I20260925 04:55:15.709414 12459 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:55:15.725004 13752 tablet_replica.cc:330] T b1e28619bee34617b532d8f73e950454 P fbde0dd6885c4cd9ab573d65bfc3231e: stopping tablet replica
I20260925 04:55:15.725598 13752 raft_consensus.cc:2229] T b1e28619bee34617b532d8f73e950454 P fbde0dd6885c4cd9ab573d65bfc3231e [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:55:15.726363 13752 raft_consensus.cc:2258] T b1e28619bee34617b532d8f73e950454 P fbde0dd6885c4cd9ab573d65bfc3231e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:55:15.727207 13750 tablet_replica.cc:330] T b1e28619bee34617b532d8f73e950454 P 2f7cb902838c4ca382524ae7baa660e4: stopping tablet replica
I20260925 04:55:15.729453 13752 ts_tablet_manager.cc:1905] T b1e28619bee34617b532d8f73e950454 P fbde0dd6885c4cd9ab573d65bfc3231e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:55:15.727672 13750 raft_consensus.cc:2229] T b1e28619bee34617b532d8f73e950454 P 2f7cb902838c4ca382524ae7baa660e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:55:15.732414 13750 raft_consensus.cc:2258] T b1e28619bee34617b532d8f73e950454 P 2f7cb902838c4ca382524ae7baa660e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:55:15.734525 13750 ts_tablet_manager.cc:1905] T b1e28619bee34617b532d8f73e950454 P 2f7cb902838c4ca382524ae7baa660e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:55:15.740279 13751 tablet_replica.cc:330] T b1e28619bee34617b532d8f73e950454 P dad67e0d8c3a4ab8adeda07f8db9a9a6: stopping tablet replica
I20260925 04:55:15.740772 13751 raft_consensus.cc:2229] T b1e28619bee34617b532d8f73e950454 P dad67e0d8c3a4ab8adeda07f8db9a9a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:55:15.741288 13751 raft_consensus.cc:2258] T b1e28619bee34617b532d8f73e950454 P dad67e0d8c3a4ab8adeda07f8db9a9a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:55:15.746225 13751 ts_tablet_manager.cc:1905] T b1e28619bee34617b532d8f73e950454 P dad67e0d8c3a4ab8adeda07f8db9a9a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:55:15.753556 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:15.683860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48874 (local address 127.6.29.189:45371)
0925 04:55:15.691713 (+ 7853us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:15.691723 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:15.699883 (+ 8160us) server_negotiation.cc:407] Connection header received
0925 04:55:15.724451 (+ 24568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:15.724462 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:15.724621 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:15.724800 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:15.735884 (+ 11084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.737504 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.742067 (+ 4563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.742683 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.742815 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:15.746571 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:15.746578 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:15.746583 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:15.746629 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:15.750100 (+ 3471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:15.751020 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:15.751026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:15.751039 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:15.751192 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:15.751636 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:15.751641 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:15.751644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:15.752064 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:15.752293 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:15.752932 (+ 639us) server_negotiation.cc:299] Negotiation successful
0925 04:55:15.753188 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7665,"thread_start_us":7518,"threads_started":1}
I20260925 04:55:15.758364 13750 ts_tablet_manager.cc:1918] T b1e28619bee34617b532d8f73e950454 P 2f7cb902838c4ca382524ae7baa660e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:55:15.758625 13750 log.cc:1196] T b1e28619bee34617b532d8f73e950454 P 2f7cb902838c4ca382524ae7baa660e4: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-1/wal/wals/b1e28619bee34617b532d8f73e950454
I20260925 04:55:15.759297 13750 ts_tablet_manager.cc:1939] T b1e28619bee34617b532d8f73e950454 P 2f7cb902838c4ca382524ae7baa660e4: Deleting consensus metadata
I20260925 04:55:15.764511 13752 ts_tablet_manager.cc:1918] T b1e28619bee34617b532d8f73e950454 P fbde0dd6885c4cd9ab573d65bfc3231e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:55:15.764731 13752 log.cc:1196] T b1e28619bee34617b532d8f73e950454 P fbde0dd6885c4cd9ab573d65bfc3231e: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-2/wal/wals/b1e28619bee34617b532d8f73e950454
I20260925 04:55:15.765400 13752 ts_tablet_manager.cc:1939] T b1e28619bee34617b532d8f73e950454 P fbde0dd6885c4cd9ab573d65bfc3231e: Deleting consensus metadata
I20260925 04:55:15.769397 13751 ts_tablet_manager.cc:1918] T b1e28619bee34617b532d8f73e950454 P dad67e0d8c3a4ab8adeda07f8db9a9a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:55:15.769619 13751 log.cc:1196] T b1e28619bee34617b532d8f73e950454 P dad67e0d8c3a4ab8adeda07f8db9a9a6: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-0/wal/wals/b1e28619bee34617b532d8f73e950454
I20260925 04:55:15.770218 13751 ts_tablet_manager.cc:1939] T b1e28619bee34617b532d8f73e950454 P dad67e0d8c3a4ab8adeda07f8db9a9a6: Deleting consensus metadata
I20260925 04:55:15.783494 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:15.700766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57556 (local address 127.6.29.188:44375)
0925 04:55:15.722172 (+ 21406us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:15.722180 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:15.754850 (+ 32670us) server_negotiation.cc:407] Connection header received
0925 04:55:15.754936 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:15.754942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:15.755027 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:15.755160 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:15.762543 (+ 7383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.763985 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.766338 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.767101 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.767246 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:15.776384 (+ 9138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:15.776392 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:15.776397 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:15.776449 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:15.780204 (+ 3755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:15.781221 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:15.781228 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:15.781232 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:15.781396 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:15.781868 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:15.781874 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:15.781877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:15.782264 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:15.782510 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:15.782882 (+ 372us) server_negotiation.cc:299] Negotiation successful
0925 04:55:15.783115 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21211,"thread_start_us":21813,"threads_started":1}
I20260925 04:55:15.881191 12413 catalog_manager.cc:4841] TS dad67e0d8c3a4ab8adeda07f8db9a9a6 (127.6.29.129:46555): tablet b1e28619bee34617b532d8f73e950454 (table test-table [id=54d8ce3b175c4205ad4f83a6ce8afce9]) successfully deleted
I20260925 04:55:15.881213 12415 catalog_manager.cc:4841] TS 2f7cb902838c4ca382524ae7baa660e4 (127.6.29.130:45893): tablet b1e28619bee34617b532d8f73e950454 (table test-table [id=54d8ce3b175c4205ad4f83a6ce8afce9]) successfully deleted
I20260925 04:55:15.881413 12414 catalog_manager.cc:4841] TS fbde0dd6885c4cd9ab573d65bfc3231e (127.6.29.131:44161): tablet b1e28619bee34617b532d8f73e950454 (table test-table [id=54d8ce3b175c4205ad4f83a6ce8afce9]) successfully deleted
I20260925 04:55:15.881981 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 12397
I20260925 04:55:15.929733 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:15.910824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48880 (local address 127.6.29.189:45371)
0925 04:55:15.911043 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:15.911049 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:15.912363 (+ 1314us) server_negotiation.cc:407] Connection header received
0925 04:55:15.912619 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:15.912625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:15.912699 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:15.912764 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:15.914912 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.915993 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.917776 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.918449 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.918561 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:15.922713 (+ 4152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:15.922720 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:15.922724 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:15.922768 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:15.926536 (+ 3768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:15.927404 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:15.927410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:15.927413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:15.927582 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:15.928064 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:15.928070 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:15.928074 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:15.928460 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:15.928734 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:15.929104 (+ 370us) server_negotiation.cc:299] Negotiation successful
0925 04:55:15.929365 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 04:55:15.931280 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:15.910831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57560 (local address 127.6.29.188:44375)
0925 04:55:15.911011 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:15.911015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:15.912202 (+ 1187us) server_negotiation.cc:407] Connection header received
0925 04:55:15.912535 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:15.912545 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:15.912654 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:15.912749 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:15.915797 (+ 3048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.917303 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.920029 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:15.920994 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:15.921128 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:15.924924 (+ 3796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:15.924931 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:15.924936 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:15.924981 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:15.928265 (+ 3284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:15.929046 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:15.929052 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:15.929055 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:15.929191 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:15.929602 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:15.929607 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:15.929612 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:15.929954 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:15.930175 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:15.930714 (+ 539us) server_negotiation.cc:299] Negotiation successful
0925 04:55:15.930964 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 04:55:16.659068 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:16.658318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.6.29.129:33855)
0925 04:55:16.658802 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:16.658915 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":274,"threads_started":1}
W20260925 04:55:16.659538 13402 heartbeater.cc:644] Failed to heartbeat to 127.6.29.190:42115 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
I20260925 04:55:16.664426 13765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:16.663495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.6.29.130:36773)
0925 04:55:16.664052 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:16.664210 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":305,"threads_started":1}
I20260925 04:55:16.664574 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:16.663495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.6.29.131:46761)
0925 04:55:16.664247 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:16.664387 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":640,"threads_started":1}
W20260925 04:55:16.664898 13551 heartbeater.cc:644] Failed to heartbeat to 127.6.29.190:42115 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
W20260925 04:55:16.664968 13692 heartbeater.cc:644] Failed to heartbeat to 127.6.29.190:42115 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
I20260925 04:55:17.176239 13767 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 932891fca8784f3cbae340352149f1fb)
I20260925 04:55:17.176527 13768 raft_consensus.cc:482] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 932891fca8784f3cbae340352149f1fb)
I20260925 04:55:17.176590 13767 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: false } } peers { permanent_uuid: "15b0f62dff3a440587c04fb55216d081" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 44375 } attrs { promote: false } }
I20260925 04:55:17.176728 13768 raft_consensus.cc:504] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: false } } peers { permanent_uuid: "15b0f62dff3a440587c04fb55216d081" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 44375 } attrs { promote: false } }
I20260925 04:55:17.177597 13768 leader_election.cc:288] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 932891fca8784f3cbae340352149f1fb (127.6.29.190:42115), 15b0f62dff3a440587c04fb55216d081 (127.6.29.188:44375)
I20260925 04:55:17.178285 13767 leader_election.cc:288] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 932891fca8784f3cbae340352149f1fb (127.6.29.190:42115), f062a4fd2dc946f99f27578e1c91d17a (127.6.29.189:45371)
I20260925 04:55:17.179220 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:17.177784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.0.0.1:47020)
0925 04:55:17.178730 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:17.178915 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":811,"thread_start_us":696,"threads_started":1}
I20260925 04:55:17.179689 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:17.178281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.0.0.1:47034)
0925 04:55:17.179132 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:17.179386 (+ 254us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":693,"threads_started":1}
W20260925 04:55:17.179661 13018 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
I20260925 04:55:17.184703 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:17.181453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.0.0.1:47044)
0925 04:55:17.184178 (+ 2725us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:17.184407 (+ 229us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2542}
W20260925 04:55:17.185045 13018 leader_election.cc:334] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 932891fca8784f3cbae340352149f1fb (127.6.29.190:42115): Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
W20260925 04:55:17.185978 13207 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
I20260925 04:55:17.188124 13771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:17.187610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.0.0.1:47054)
0925 04:55:17.187818 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:17.187951 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 04:55:17.188351 13207 leader_election.cc:334] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 932891fca8784f3cbae340352149f1fb (127.6.29.190:42115): Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
I20260925 04:55:17.191710 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:17.178637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:45371 (local address 127.0.0.1:48882)
0925 04:55:17.180067 (+ 1430us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:17.180090 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:17.180281 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:17.186434 (+ 6153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:17.186439 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:17.186483 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:55:17.186765 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:17.186778 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:17.188473 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:17.188479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:17.189805 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:17.189818 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:17.190927 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:17.190933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:17.191097 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:17.191124 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 04:55:17.191231 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 04:55:17.191371 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1301,"thread_start_us":202,"threads_started":1}
I20260925 04:55:17.191797 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:17.178732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48882 (local address 127.6.29.189:45371)
0925 04:55:17.186107 (+ 7375us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:17.186116 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:17.186137 (+ 21us) server_negotiation.cc:407] Connection header received
0925 04:55:17.186207 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:17.186212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:17.186284 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:17.186396 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:55:17.186992 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:17.188314 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:17.189951 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:17.190743 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:17.190894 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:17.191004 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:17.191264 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 04:55:17.191394 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":7258,"thread_start_us":7110,"threads_started":1}
I20260925 04:55:17.191946 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:17.177812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:44375 (local address 127.0.0.1:57574)
0925 04:55:17.180936 (+ 3124us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:17.180959 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:17.181121 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:17.186705 (+ 5584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:17.186711 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:17.186751 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:55:17.187107 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:17.187128 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:17.188682 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:17.188688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:17.189920 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:17.189932 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:17.191059 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:17.191065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:17.191246 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:17.191269 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 04:55:17.191406 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 04:55:17.191553 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2984,"mutex_wait_us":51,"thread_start_us":353,"threads_started":1}
I20260925 04:55:17.192011 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:17.184415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57574 (local address 127.6.29.188:44375)
0925 04:55:17.186376 (+ 1961us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:17.186383 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:17.186400 (+ 17us) server_negotiation.cc:407] Connection header received
0925 04:55:17.186442 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:17.186446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:17.186526 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:17.187042 (+ 516us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:55:17.187341 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:17.188567 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:17.190088 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:17.190863 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:17.191023 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:17.191127 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:17.191430 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 04:55:17.191559 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1827,"spinlock_wait_cycles":19840,"thread_start_us":1713,"threads_started":1}
I20260925 04:55:17.192571 13044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15b0f62dff3a440587c04fb55216d081" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f062a4fd2dc946f99f27578e1c91d17a" is_pre_election: true
I20260925 04:55:17.192553 13233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f062a4fd2dc946f99f27578e1c91d17a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "15b0f62dff3a440587c04fb55216d081" is_pre_election: true
I20260925 04:55:17.192909 13233 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f062a4fd2dc946f99f27578e1c91d17a in term 1.
I20260925 04:55:17.192890 13044 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15b0f62dff3a440587c04fb55216d081 in term 1.
I20260925 04:55:17.193369 13210 leader_election.cc:302] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 15b0f62dff3a440587c04fb55216d081, f062a4fd2dc946f99f27578e1c91d17a; no voters: 932891fca8784f3cbae340352149f1fb
I20260925 04:55:17.193454 13020 leader_election.cc:302] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 15b0f62dff3a440587c04fb55216d081, f062a4fd2dc946f99f27578e1c91d17a; no voters: 932891fca8784f3cbae340352149f1fb
I20260925 04:55:17.193795 13767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 04:55:17.193799 13768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 04:55:17.193920 13768 raft_consensus.cc:482] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 932891fca8784f3cbae340352149f1fb)
I20260925 04:55:17.193939 13767 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 932891fca8784f3cbae340352149f1fb)
I20260925 04:55:17.194031 13768 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [term 1 FOLLOWER]: Advancing to term 2
I20260925 04:55:17.194031 13767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [term 1 FOLLOWER]: Advancing to term 2
I20260925 04:55:17.197166 13767 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: false } } peers { permanent_uuid: "15b0f62dff3a440587c04fb55216d081" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 44375 } attrs { promote: false } }
I20260925 04:55:17.197171 13768 raft_consensus.cc:504] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: false } } peers { permanent_uuid: "15b0f62dff3a440587c04fb55216d081" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 44375 } attrs { promote: false } }
I20260925 04:55:17.197888 13767 leader_election.cc:288] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [CANDIDATE]: Term 2 election: Requested vote from peers 932891fca8784f3cbae340352149f1fb (127.6.29.190:42115), f062a4fd2dc946f99f27578e1c91d17a (127.6.29.189:45371)
I20260925 04:55:17.198308 13768 leader_election.cc:288] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [CANDIDATE]: Term 2 election: Requested vote from peers 932891fca8784f3cbae340352149f1fb (127.6.29.190:42115), 15b0f62dff3a440587c04fb55216d081 (127.6.29.188:44375)
I20260925 04:55:17.198498 13044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "15b0f62dff3a440587c04fb55216d081" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f062a4fd2dc946f99f27578e1c91d17a"
I20260925 04:55:17.198702 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:17.198135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.0.0.1:47070)
0925 04:55:17.198342 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:17.198438 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 04:55:17.198836 13044 raft_consensus.cc:2379] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 15b0f62dff3a440587c04fb55216d081 in current term 2: Already voted for candidate f062a4fd2dc946f99f27578e1c91d17a in this term.
I20260925 04:55:17.198820 13233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f062a4fd2dc946f99f27578e1c91d17a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "15b0f62dff3a440587c04fb55216d081"
I20260925 04:55:17.199095 13233 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f062a4fd2dc946f99f27578e1c91d17a in current term 2: Already voted for candidate 15b0f62dff3a440587c04fb55216d081 in this term.
I20260925 04:55:17.199501 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:17.198612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.0.0.1:47078)
0925 04:55:17.199286 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:17.199358 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":553}
I20260925 04:55:17.200214 13769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:17.199739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.0.0.1:47084)
0925 04:55:17.199987 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:17.200050 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
W20260925 04:55:17.200377 13018 leader_election.cc:334] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 932891fca8784f3cbae340352149f1fb (127.6.29.190:42115): Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
I20260925 04:55:17.200531 13772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:17.200149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.0.0.1:47096)
0925 04:55:17.200287 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:17.200386 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 04:55:17.200485 13018 leader_election.cc:302] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f062a4fd2dc946f99f27578e1c91d17a; no voters: 15b0f62dff3a440587c04fb55216d081, 932891fca8784f3cbae340352149f1fb
W20260925 04:55:17.200716 13207 leader_election.cc:334] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 932891fca8784f3cbae340352149f1fb (127.6.29.190:42115): Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
I20260925 04:55:17.200768 13768 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 04:55:17.200922 13207 leader_election.cc:302] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 15b0f62dff3a440587c04fb55216d081; no voters: 932891fca8784f3cbae340352149f1fb, f062a4fd2dc946f99f27578e1c91d17a
I20260925 04:55:17.201246 13767 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 04:55:17.661201 13777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:17.660296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.6.29.129:34759)
0925 04:55:17.660853 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:17.660957 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":294,"threads_started":1}
I20260925 04:55:17.668725 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:17.666230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.6.29.131:51475)
0925 04:55:17.668356 (+ 2126us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:17.668467 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1982,"thread_start_us":1876,"threads_started":1}
I20260925 04:55:17.668974 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:17.666146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.6.29.130:52095)
0925 04:55:17.668676 (+ 2530us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:17.668752 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2410,"thread_start_us":2306,"threads_started":1}
I20260925 04:55:18.662947 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:18.662182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.6.29.129:52581)
0925 04:55:18.662623 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:18.662734 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":230,"threads_started":1}
I20260925 04:55:18.670223 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:18.669544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.6.29.131:55933)
0925 04:55:18.669960 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:18.670040 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":222,"threads_started":1}
I20260925 04:55:18.670509 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:18.669818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.6.29.130:59271)
0925 04:55:18.670229 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:18.670312 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":223,"threads_started":1}
I20260925 04:55:19.017874 13783 raft_consensus.cc:482] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:55:19.018141 13783 raft_consensus.cc:504] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: false } } peers { permanent_uuid: "15b0f62dff3a440587c04fb55216d081" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 44375 } attrs { promote: false } }
I20260925 04:55:19.018898 13783 leader_election.cc:288] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 932891fca8784f3cbae340352149f1fb (127.6.29.190:42115), 15b0f62dff3a440587c04fb55216d081 (127.6.29.188:44375)
I20260925 04:55:19.019727 13233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f062a4fd2dc946f99f27578e1c91d17a" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "15b0f62dff3a440587c04fb55216d081" is_pre_election: true
I20260925 04:55:19.019994 13784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:19.019056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.0.0.1:47346)
0925 04:55:19.019579 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:19.019715 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":229,"threads_started":1}
I20260925 04:55:19.020041 13233 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f062a4fd2dc946f99f27578e1c91d17a in term 2.
I20260925 04:55:19.020623 13020 leader_election.cc:302] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15b0f62dff3a440587c04fb55216d081, f062a4fd2dc946f99f27578e1c91d17a; no voters:
I20260925 04:55:19.020927 13783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 04:55:19.021042 13783 raft_consensus.cc:482] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:55:19.021188 13783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [term 2 FOLLOWER]: Advancing to term 3
I20260925 04:55:19.021733 13784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:19.021298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.0.0.1:47348)
0925 04:55:19.021540 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:19.021618 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260925 04:55:19.022012 13018 leader_election.cc:334] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 932891fca8784f3cbae340352149f1fb (127.6.29.190:42115): Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
I20260925 04:55:19.023540 13783 raft_consensus.cc:504] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: false } } peers { permanent_uuid: "15b0f62dff3a440587c04fb55216d081" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 44375 } attrs { promote: false } }
I20260925 04:55:19.024173 13783 leader_election.cc:288] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [CANDIDATE]: Term 3 election: Requested vote from peers 932891fca8784f3cbae340352149f1fb (127.6.29.190:42115), 15b0f62dff3a440587c04fb55216d081 (127.6.29.188:44375)
I20260925 04:55:19.024711 13784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:19.024276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.0.0.1:47364)
0925 04:55:19.024442 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:19.024511 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 04:55:19.024717 13233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f062a4fd2dc946f99f27578e1c91d17a" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "15b0f62dff3a440587c04fb55216d081"
I20260925 04:55:19.024997 13233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [term 2 FOLLOWER]: Advancing to term 3
I20260925 04:55:19.025687 13784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:19.025336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.0.0.1:47366)
0925 04:55:19.025485 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:19.025564 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 04:55:19.025859 13018 leader_election.cc:334] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 932891fca8784f3cbae340352149f1fb (127.6.29.190:42115): Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
I20260925 04:55:19.027283 13233 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f062a4fd2dc946f99f27578e1c91d17a in term 3.
I20260925 04:55:19.027752 13020 leader_election.cc:302] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 15b0f62dff3a440587c04fb55216d081, f062a4fd2dc946f99f27578e1c91d17a; no voters: 932891fca8784f3cbae340352149f1fb
I20260925 04:55:19.028079 13783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [term 3 FOLLOWER]: Leader election won for term 3
I20260925 04:55:19.028508 13783 raft_consensus.cc:686] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [term 3 LEADER]: Becoming Leader. State: Replica: f062a4fd2dc946f99f27578e1c91d17a, State: Running, Role: LEADER
I20260925 04:55:19.028882 13783 consensus_queue.cc:237] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: false } } peers { permanent_uuid: "15b0f62dff3a440587c04fb55216d081" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 44375 } attrs { promote: false } }
I20260925 04:55:19.030589 13786 sys_catalog.cc:455] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f062a4fd2dc946f99f27578e1c91d17a. Latest consensus state: current_term: 3 leader_uuid: "f062a4fd2dc946f99f27578e1c91d17a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: false } } peers { permanent_uuid: "15b0f62dff3a440587c04fb55216d081" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 44375 } attrs { promote: false } } }
I20260925 04:55:19.030969 13786 sys_catalog.cc:458] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [sys.catalog]: This master's current role is: LEADER
I20260925 04:55:19.031569 13788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:55:19.035035 13788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:55:19.035939 13788 catalog_manager.cc:1259] Loaded cluster ID: f48087faeea44abca781b012f0655bbc
I20260925 04:55:19.036031 13788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:55:19.037163 13788 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:55:19.038009 13788 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a: Loaded TSK: 0
I20260925 04:55:19.038915 13788 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:55:19.124425 13033 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48880:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:55:19.131412 13033 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:55:19.135752 13784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:19.135276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.0.0.1:47408)
0925 04:55:19.135480 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:19.135574 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 04:55:19.135797 13233 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [term 3 FOLLOWER]: Refusing update from remote peer f062a4fd2dc946f99f27578e1c91d17a: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260925 04:55:19.136576 13786 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [LEADER]: Connected to new peer: Peer: permanent_uuid: "15b0f62dff3a440587c04fb55216d081" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 44375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260925 04:55:19.136825 13784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:19.136471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.0.0.1:47416)
0925 04:55:19.136624 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:19.136692 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 04:55:19.137120 13018 consensus_peers.cc:487] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a -> Peer 932891fca8784f3cbae340352149f1fb (127.6.29.190:42115): Couldn't send request to peer 932891fca8784f3cbae340352149f1fb. Status: Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 04:55:19.139446 13790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f062a4fd2dc946f99f27578e1c91d17a. Latest consensus state: current_term: 3 leader_uuid: "f062a4fd2dc946f99f27578e1c91d17a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: false } } peers { permanent_uuid: "15b0f62dff3a440587c04fb55216d081" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 44375 } attrs { promote: false } } }
I20260925 04:55:19.139827 13790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:55:19.144477 13790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "f062a4fd2dc946f99f27578e1c91d17a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: false } } peers { permanent_uuid: "15b0f62dff3a440587c04fb55216d081" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 44375 } attrs { promote: false } } }
I20260925 04:55:19.144929 13790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15b0f62dff3a440587c04fb55216d081 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:55:19.146014 13795 sys_catalog.cc:455] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "f062a4fd2dc946f99f27578e1c91d17a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "932891fca8784f3cbae340352149f1fb" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 42115 } } peers { permanent_uuid: "f062a4fd2dc946f99f27578e1c91d17a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 45371 } attrs { promote: false } } peers { permanent_uuid: "15b0f62dff3a440587c04fb55216d081" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 44375 } attrs { promote: false } } }
I20260925 04:55:19.146454 13795 sys_catalog.cc:458] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a [sys.catalog]: This master's current role is: LEADER
I20260925 04:55:19.166010 13784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:19.153802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:45893 (local address 127.0.0.1:43192)
0925 04:55:19.154033 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:19.154052 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:19.154171 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:19.155755 (+ 1584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:19.155791 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:19.155815 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:55:19.156184 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:19.156201 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:19.158266 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:19.158276 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:19.161125 (+ 2849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:19.161147 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:19.164912 (+ 3765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:19.164922 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:19.165205 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:19.165265 (+ 60us) client_negotiation.cc:769] Sending connection context
0925 04:55:19.165347 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 04:55:19.165554 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260925 04:55:19.167235 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:19.154018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43192 (local address 127.6.29.130:45893)
0925 04:55:19.155346 (+ 1328us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:19.155352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:19.155370 (+ 18us) server_negotiation.cc:407] Connection header received
0925 04:55:19.155472 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:19.155479 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:19.155587 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:19.156360 (+ 773us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:55:19.156612 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:19.158077 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:19.162560 (+ 4483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:19.164654 (+ 2094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:19.166326 (+ 1672us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:19.166481 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:19.166546 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 04:55:19.166754 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1183,"thread_start_us":1053,"threads_started":1}
I20260925 04:55:19.168324 13485 tablet_service.cc:1467] Processing CreateTablet for tablet 66ac8ff5456b400fba4189b906d9ded7 (DEFAULT_TABLE table=test-table [id=523712709bbb4a309922220aaa288a41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:55:19.168951 13485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66ac8ff5456b400fba4189b906d9ded7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:55:19.173980 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:19.154576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45130 (local address 127.6.29.131:44161)
0925 04:55:19.168013 (+ 13437us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:19.168017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:19.168033 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:55:19.168090 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:19.168093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:19.168158 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:19.168280 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:55:19.168829 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:19.169592 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:19.171547 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:19.172882 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:19.173024 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:19.173168 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:19.173485 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 04:55:19.173631 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13328,"thread_start_us":13827,"threads_started":1}
I20260925 04:55:19.174227 13797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:19.154452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.131:44161 (local address 127.0.0.1:45130)
0925 04:55:19.155173 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:19.155192 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:19.155304 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:19.168301 (+ 12997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:19.168305 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:19.168333 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:55:19.168662 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:19.168670 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:19.169727 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:19.169734 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:19.171338 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:19.171358 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:19.173119 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:19.173134 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:19.173292 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:19.173361 (+ 69us) client_negotiation.cc:769] Sending connection context
0925 04:55:19.173454 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 04:55:19.173653 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":410,"threads_started":1}
I20260925 04:55:19.175119 13626 tablet_service.cc:1467] Processing CreateTablet for tablet 66ac8ff5456b400fba4189b906d9ded7 (DEFAULT_TABLE table=test-table [id=523712709bbb4a309922220aaa288a41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:55:19.175606 13626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66ac8ff5456b400fba4189b906d9ded7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:55:19.178875 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:19.155343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:46555 (local address 127.0.0.1:54686)
0925 04:55:19.155903 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:19.155926 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:19.156040 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:19.171832 (+ 15792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:19.171839 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:19.171871 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:55:19.172211 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:19.172222 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:19.173690 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:19.173698 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:19.176221 (+ 2523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:19.176235 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:19.177983 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:19.177992 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:19.178182 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:19.178215 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 04:55:19.178297 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 04:55:19.178431 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":283,"threads_started":1}
I20260925 04:55:19.179649 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:19.155439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54686 (local address 127.6.29.129:46555)
0925 04:55:19.171462 (+ 16023us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:19.171469 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:19.171490 (+ 21us) server_negotiation.cc:407] Connection header received
0925 04:55:19.171573 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:19.171578 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:19.171655 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:19.171791 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:55:19.172362 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:19.173486 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:19.176385 (+ 2899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:19.177594 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:19.177729 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:19.177835 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:19.179216 (+ 1381us) server_negotiation.cc:299] Negotiation successful
0925 04:55:19.179370 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15883,"thread_start_us":16463,"threads_started":1}
I20260925 04:55:19.180598 13801 tablet_bootstrap.cc:492] T 66ac8ff5456b400fba4189b906d9ded7 P 2f7cb902838c4ca382524ae7baa660e4: Bootstrap starting.
I20260925 04:55:19.180823 13336 tablet_service.cc:1467] Processing CreateTablet for tablet 66ac8ff5456b400fba4189b906d9ded7 (DEFAULT_TABLE table=test-table [id=523712709bbb4a309922220aaa288a41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:55:19.181280 13336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66ac8ff5456b400fba4189b906d9ded7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:55:19.182092 13801 tablet_bootstrap.cc:654] T 66ac8ff5456b400fba4189b906d9ded7 P 2f7cb902838c4ca382524ae7baa660e4: Neither blocks nor log segments found. Creating new log.
I20260925 04:55:19.182717 13802 tablet_bootstrap.cc:492] T 66ac8ff5456b400fba4189b906d9ded7 P fbde0dd6885c4cd9ab573d65bfc3231e: Bootstrap starting.
I20260925 04:55:19.184413 13802 tablet_bootstrap.cc:654] T 66ac8ff5456b400fba4189b906d9ded7 P fbde0dd6885c4cd9ab573d65bfc3231e: Neither blocks nor log segments found. Creating new log.
I20260925 04:55:19.184491 13801 tablet_bootstrap.cc:492] T 66ac8ff5456b400fba4189b906d9ded7 P 2f7cb902838c4ca382524ae7baa660e4: No bootstrap required, opened a new log
I20260925 04:55:19.184697 13801 ts_tablet_manager.cc:1397] T 66ac8ff5456b400fba4189b906d9ded7 P 2f7cb902838c4ca382524ae7baa660e4: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20260925 04:55:19.185640 13801 raft_consensus.cc:348] T 66ac8ff5456b400fba4189b906d9ded7 P 2f7cb902838c4ca382524ae7baa660e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7cb902838c4ca382524ae7baa660e4" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45893 } } peers { permanent_uuid: "fbde0dd6885c4cd9ab573d65bfc3231e" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 44161 } } peers { permanent_uuid: "dad67e0d8c3a4ab8adeda07f8db9a9a6" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 46555 } }
I20260925 04:55:19.186024 13801 raft_consensus.cc:374] T 66ac8ff5456b400fba4189b906d9ded7 P 2f7cb902838c4ca382524ae7baa660e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:55:19.186316 13801 raft_consensus.cc:729] T 66ac8ff5456b400fba4189b906d9ded7 P 2f7cb902838c4ca382524ae7baa660e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f7cb902838c4ca382524ae7baa660e4, State: Initialized, Role: FOLLOWER
I20260925 04:55:19.186718 13801 consensus_queue.cc:260] T 66ac8ff5456b400fba4189b906d9ded7 P 2f7cb902838c4ca382524ae7baa660e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7cb902838c4ca382524ae7baa660e4" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45893 } } peers { permanent_uuid: "fbde0dd6885c4cd9ab573d65bfc3231e" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 44161 } } peers { permanent_uuid: "dad67e0d8c3a4ab8adeda07f8db9a9a6" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 46555 } }
I20260925 04:55:19.187654 13805 tablet_bootstrap.cc:492] T 66ac8ff5456b400fba4189b906d9ded7 P dad67e0d8c3a4ab8adeda07f8db9a9a6: Bootstrap starting.
I20260925 04:55:19.189249 13805 tablet_bootstrap.cc:654] T 66ac8ff5456b400fba4189b906d9ded7 P dad67e0d8c3a4ab8adeda07f8db9a9a6: Neither blocks nor log segments found. Creating new log.
I20260925 04:55:19.191176 13805 tablet_bootstrap.cc:492] T 66ac8ff5456b400fba4189b906d9ded7 P dad67e0d8c3a4ab8adeda07f8db9a9a6: No bootstrap required, opened a new log
I20260925 04:55:19.191342 13805 ts_tablet_manager.cc:1397] T 66ac8ff5456b400fba4189b906d9ded7 P dad67e0d8c3a4ab8adeda07f8db9a9a6: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 04:55:19.192173 13805 raft_consensus.cc:348] T 66ac8ff5456b400fba4189b906d9ded7 P dad67e0d8c3a4ab8adeda07f8db9a9a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7cb902838c4ca382524ae7baa660e4" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45893 } } peers { permanent_uuid: "fbde0dd6885c4cd9ab573d65bfc3231e" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 44161 } } peers { permanent_uuid: "dad67e0d8c3a4ab8adeda07f8db9a9a6" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 46555 } }
I20260925 04:55:19.192461 13805 raft_consensus.cc:374] T 66ac8ff5456b400fba4189b906d9ded7 P dad67e0d8c3a4ab8adeda07f8db9a9a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:55:19.192595 13805 raft_consensus.cc:729] T 66ac8ff5456b400fba4189b906d9ded7 P dad67e0d8c3a4ab8adeda07f8db9a9a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dad67e0d8c3a4ab8adeda07f8db9a9a6, State: Initialized, Role: FOLLOWER
I20260925 04:55:19.192896 13805 consensus_queue.cc:260] T 66ac8ff5456b400fba4189b906d9ded7 P dad67e0d8c3a4ab8adeda07f8db9a9a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7cb902838c4ca382524ae7baa660e4" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45893 } } peers { permanent_uuid: "fbde0dd6885c4cd9ab573d65bfc3231e" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 44161 } } peers { permanent_uuid: "dad67e0d8c3a4ab8adeda07f8db9a9a6" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 46555 } }
I20260925 04:55:19.193396 13802 tablet_bootstrap.cc:492] T 66ac8ff5456b400fba4189b906d9ded7 P fbde0dd6885c4cd9ab573d65bfc3231e: No bootstrap required, opened a new log
I20260925 04:55:19.194120 13802 ts_tablet_manager.cc:1397] T 66ac8ff5456b400fba4189b906d9ded7 P fbde0dd6885c4cd9ab573d65bfc3231e: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260925 04:55:19.195209 13801 ts_tablet_manager.cc:1428] T 66ac8ff5456b400fba4189b906d9ded7 P 2f7cb902838c4ca382524ae7baa660e4: Time spent starting tablet: real 0.010s user 0.001s sys 0.007s
I20260925 04:55:19.195266 13802 raft_consensus.cc:348] T 66ac8ff5456b400fba4189b906d9ded7 P fbde0dd6885c4cd9ab573d65bfc3231e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7cb902838c4ca382524ae7baa660e4" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45893 } } peers { permanent_uuid: "fbde0dd6885c4cd9ab573d65bfc3231e" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 44161 } } peers { permanent_uuid: "dad67e0d8c3a4ab8adeda07f8db9a9a6" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 46555 } }
I20260925 04:55:19.195605 13802 raft_consensus.cc:374] T 66ac8ff5456b400fba4189b906d9ded7 P fbde0dd6885c4cd9ab573d65bfc3231e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:55:19.195730 13802 raft_consensus.cc:729] T 66ac8ff5456b400fba4189b906d9ded7 P fbde0dd6885c4cd9ab573d65bfc3231e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbde0dd6885c4cd9ab573d65bfc3231e, State: Initialized, Role: FOLLOWER
I20260925 04:55:19.195955 13805 ts_tablet_manager.cc:1428] T 66ac8ff5456b400fba4189b906d9ded7 P dad67e0d8c3a4ab8adeda07f8db9a9a6: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260925 04:55:19.196094 13802 consensus_queue.cc:260] T 66ac8ff5456b400fba4189b906d9ded7 P fbde0dd6885c4cd9ab573d65bfc3231e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7cb902838c4ca382524ae7baa660e4" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45893 } } peers { permanent_uuid: "fbde0dd6885c4cd9ab573d65bfc3231e" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 44161 } } peers { permanent_uuid: "dad67e0d8c3a4ab8adeda07f8db9a9a6" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 46555 } }
I20260925 04:55:19.196923 13802 ts_tablet_manager.cc:1428] T 66ac8ff5456b400fba4189b906d9ded7 P fbde0dd6885c4cd9ab573d65bfc3231e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 04:55:19.201085 13693 heartbeater.cc:503] Master 127.6.29.189:45371 was elected leader, sending a full tablet report...
I20260925 04:55:19.201395 13552 heartbeater.cc:503] Master 127.6.29.189:45371 was elected leader, sending a full tablet report...
I20260925 04:55:19.203969 13403 heartbeater.cc:503] Master 127.6.29.189:45371 was elected leader, sending a full tablet report...
I20260925 04:55:19.210464 13809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:19.194494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.6.29.130:48335)
0925 04:55:19.209771 (+ 15277us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:19.210054 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":319,"threads_started":1}
I20260925 04:55:19.210813 13811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:19.197013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.6.29.129:47635)
0925 04:55:19.210417 (+ 13404us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:19.210559 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13279,"threads_started":1}
I20260925 04:55:19.211889 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:19.197761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.6.29.131:56245)
0925 04:55:19.211415 (+ 13654us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:19.211537 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13528,"thread_start_us":13385,"threads_started":1}
I20260925 04:55:19.212148 13811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:19.211681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.6.29.129:45579)
0925 04:55:19.211908 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:19.211996 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260925 04:55:19.212481 13813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:19.211339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.6.29.130:53895)
0925 04:55:19.211928 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:19.212197 (+ 269us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":317,"threads_started":1}
I20260925 04:55:19.364959 13810 raft_consensus.cc:482] T 66ac8ff5456b400fba4189b906d9ded7 P fbde0dd6885c4cd9ab573d65bfc3231e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:55:19.365262 13810 raft_consensus.cc:504] T 66ac8ff5456b400fba4189b906d9ded7 P fbde0dd6885c4cd9ab573d65bfc3231e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7cb902838c4ca382524ae7baa660e4" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45893 } } peers { permanent_uuid: "fbde0dd6885c4cd9ab573d65bfc3231e" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 44161 } } peers { permanent_uuid: "dad67e0d8c3a4ab8adeda07f8db9a9a6" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 46555 } }
I20260925 04:55:19.366060 13810 leader_election.cc:288] T 66ac8ff5456b400fba4189b906d9ded7 P fbde0dd6885c4cd9ab573d65bfc3231e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f7cb902838c4ca382524ae7baa660e4 (127.6.29.130:45893), dad67e0d8c3a4ab8adeda07f8db9a9a6 (127.6.29.129:46555)
I20260925 04:55:19.366642 13356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66ac8ff5456b400fba4189b906d9ded7" candidate_uuid: "fbde0dd6885c4cd9ab573d65bfc3231e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dad67e0d8c3a4ab8adeda07f8db9a9a6" is_pre_election: true
I20260925 04:55:19.366643 13505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66ac8ff5456b400fba4189b906d9ded7" candidate_uuid: "fbde0dd6885c4cd9ab573d65bfc3231e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f7cb902838c4ca382524ae7baa660e4" is_pre_election: true
I20260925 04:55:19.366887 13505 raft_consensus.cc:2454] T 66ac8ff5456b400fba4189b906d9ded7 P 2f7cb902838c4ca382524ae7baa660e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbde0dd6885c4cd9ab573d65bfc3231e in term 0.
I20260925 04:55:19.366881 13356 raft_consensus.cc:2454] T 66ac8ff5456b400fba4189b906d9ded7 P dad67e0d8c3a4ab8adeda07f8db9a9a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbde0dd6885c4cd9ab573d65bfc3231e in term 0.
I20260925 04:55:19.367414 13579 leader_election.cc:302] T 66ac8ff5456b400fba4189b906d9ded7 P fbde0dd6885c4cd9ab573d65bfc3231e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2f7cb902838c4ca382524ae7baa660e4, dad67e0d8c3a4ab8adeda07f8db9a9a6, fbde0dd6885c4cd9ab573d65bfc3231e; no voters:
I20260925 04:55:19.367720 13810 raft_consensus.cc:2781] T 66ac8ff5456b400fba4189b906d9ded7 P fbde0dd6885c4cd9ab573d65bfc3231e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:55:19.367878 13810 raft_consensus.cc:482] T 66ac8ff5456b400fba4189b906d9ded7 P fbde0dd6885c4cd9ab573d65bfc3231e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:55:19.367985 13810 raft_consensus.cc:3037] T 66ac8ff5456b400fba4189b906d9ded7 P fbde0dd6885c4cd9ab573d65bfc3231e [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:55:19.370082 13810 raft_consensus.cc:504] T 66ac8ff5456b400fba4189b906d9ded7 P fbde0dd6885c4cd9ab573d65bfc3231e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7cb902838c4ca382524ae7baa660e4" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45893 } } peers { permanent_uuid: "fbde0dd6885c4cd9ab573d65bfc3231e" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 44161 } } peers { permanent_uuid: "dad67e0d8c3a4ab8adeda07f8db9a9a6" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 46555 } }
I20260925 04:55:19.370683 13810 leader_election.cc:288] T 66ac8ff5456b400fba4189b906d9ded7 P fbde0dd6885c4cd9ab573d65bfc3231e [CANDIDATE]: Term 1 election: Requested vote from peers 2f7cb902838c4ca382524ae7baa660e4 (127.6.29.130:45893), dad67e0d8c3a4ab8adeda07f8db9a9a6 (127.6.29.129:46555)
I20260925 04:55:19.371228 13356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66ac8ff5456b400fba4189b906d9ded7" candidate_uuid: "fbde0dd6885c4cd9ab573d65bfc3231e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dad67e0d8c3a4ab8adeda07f8db9a9a6"
I20260925 04:55:19.371218 13505 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66ac8ff5456b400fba4189b906d9ded7" candidate_uuid: "fbde0dd6885c4cd9ab573d65bfc3231e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f7cb902838c4ca382524ae7baa660e4"
I20260925 04:55:19.371433 13356 raft_consensus.cc:3037] T 66ac8ff5456b400fba4189b906d9ded7 P dad67e0d8c3a4ab8adeda07f8db9a9a6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:55:19.371433 13505 raft_consensus.cc:3037] T 66ac8ff5456b400fba4189b906d9ded7 P 2f7cb902838c4ca382524ae7baa660e4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:55:19.374485 13356 raft_consensus.cc:2454] T 66ac8ff5456b400fba4189b906d9ded7 P dad67e0d8c3a4ab8adeda07f8db9a9a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbde0dd6885c4cd9ab573d65bfc3231e in term 1.
I20260925 04:55:19.374493 13505 raft_consensus.cc:2454] T 66ac8ff5456b400fba4189b906d9ded7 P 2f7cb902838c4ca382524ae7baa660e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbde0dd6885c4cd9ab573d65bfc3231e in term 1.
I20260925 04:55:19.374922 13581 leader_election.cc:302] T 66ac8ff5456b400fba4189b906d9ded7 P fbde0dd6885c4cd9ab573d65bfc3231e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2f7cb902838c4ca382524ae7baa660e4, dad67e0d8c3a4ab8adeda07f8db9a9a6, fbde0dd6885c4cd9ab573d65bfc3231e; no voters:
I20260925 04:55:19.375268 13810 raft_consensus.cc:2781] T 66ac8ff5456b400fba4189b906d9ded7 P fbde0dd6885c4cd9ab573d65bfc3231e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:55:19.375869 13810 raft_consensus.cc:686] T 66ac8ff5456b400fba4189b906d9ded7 P fbde0dd6885c4cd9ab573d65bfc3231e [term 1 LEADER]: Becoming Leader. State: Replica: fbde0dd6885c4cd9ab573d65bfc3231e, State: Running, Role: LEADER
I20260925 04:55:19.376220 13810 consensus_queue.cc:237] T 66ac8ff5456b400fba4189b906d9ded7 P fbde0dd6885c4cd9ab573d65bfc3231e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7cb902838c4ca382524ae7baa660e4" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45893 } } peers { permanent_uuid: "fbde0dd6885c4cd9ab573d65bfc3231e" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 44161 } } peers { permanent_uuid: "dad67e0d8c3a4ab8adeda07f8db9a9a6" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 46555 } }
I20260925 04:55:19.377071 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:19.376631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.6.29.131:53335)
0925 04:55:19.376832 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:19.376926 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 04:55:19.384779 13033 catalog_manager.cc:5495] T 66ac8ff5456b400fba4189b906d9ded7 P fbde0dd6885c4cd9ab573d65bfc3231e reported cstate change: term changed from 0 to 1, leader changed from <none> to fbde0dd6885c4cd9ab573d65bfc3231e (127.6.29.131). New cstate: current_term: 1 leader_uuid: "fbde0dd6885c4cd9ab573d65bfc3231e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f7cb902838c4ca382524ae7baa660e4" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbde0dd6885c4cd9ab573d65bfc3231e" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 44161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dad67e0d8c3a4ab8adeda07f8db9a9a6" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 46555 } health_report { overall_health: UNKNOWN } } }
I20260925 04:55:19.411623 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:19.406785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53516 (local address 127.6.29.131:44161)
0925 04:55:19.406990 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:19.406996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:19.407015 (+ 19us) server_negotiation.cc:407] Connection header received
0925 04:55:19.407212 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:19.407216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:19.407278 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:19.407371 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:55:19.407791 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:19.408488 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:19.409852 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:19.410268 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:19.410403 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:19.410838 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:55:19.410939 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:55:19.411041 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:19.411218 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 04:55:19.411296 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 04:55:19.414796 13356 raft_consensus.cc:1261] T 66ac8ff5456b400fba4189b906d9ded7 P dad67e0d8c3a4ab8adeda07f8db9a9a6 [term 1 FOLLOWER]: Refusing update from remote peer fbde0dd6885c4cd9ab573d65bfc3231e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:55:19.414832 13505 raft_consensus.cc:1261] T 66ac8ff5456b400fba4189b906d9ded7 P 2f7cb902838c4ca382524ae7baa660e4 [term 1 FOLLOWER]: Refusing update from remote peer fbde0dd6885c4cd9ab573d65bfc3231e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:55:19.415436 13810 consensus_queue.cc:1035] T 66ac8ff5456b400fba4189b906d9ded7 P fbde0dd6885c4cd9ab573d65bfc3231e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f7cb902838c4ca382524ae7baa660e4" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:55:19.415951 13814 consensus_queue.cc:1035] T 66ac8ff5456b400fba4189b906d9ded7 P fbde0dd6885c4cd9ab573d65bfc3231e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dad67e0d8c3a4ab8adeda07f8db9a9a6" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 46555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:55:19.417999 13813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:19.417346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.6.29.130:51381)
0925 04:55:19.417747 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:19.417840 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263}
I20260925 04:55:19.418104 13811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:19.417681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.6.29.129:55759)
0925 04:55:19.417835 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:19.417912 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 04:55:19.429682 13811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:19.429046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.6.29.129:60607)
0925 04:55:19.429387 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:19.429489 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212}
I20260925 04:55:19.432731 13813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:19.432040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.6.29.130:37351)
0925 04:55:19.432343 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:19.432472 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207}
I20260925 04:55:19.435432 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:19.434837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.6.29.131:53579)
0925 04:55:19.435147 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:19.435257 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203}
I20260925 04:55:19.439167 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:19.438705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.6.29.131:57617)
0925 04:55:19.438898 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:19.438981 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 04:55:19.442800 13033 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:55:19.443050 13033 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:55:19.448437 13033 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f062a4fd2dc946f99f27578e1c91d17a: Sending DeleteTablet for 3 replicas of tablet 66ac8ff5456b400fba4189b906d9ded7
I20260925 04:55:19.449455 13626 tablet_service.cc:1514] Processing DeleteTablet for tablet 66ac8ff5456b400fba4189b906d9ded7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:55:19 UTC) from {username='oryx', principal='oryx/127.6.29.189@KRBTEST.COM'} at 127.0.0.1:45130
I20260925 04:55:19.449528 13485 tablet_service.cc:1514] Processing DeleteTablet for tablet 66ac8ff5456b400fba4189b906d9ded7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:55:19 UTC) from {username='oryx', principal='oryx/127.6.29.189@KRBTEST.COM'} at 127.0.0.1:43192
I20260925 04:55:19.449970 13336 tablet_service.cc:1514] Processing DeleteTablet for tablet 66ac8ff5456b400fba4189b906d9ded7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:55:19 UTC) from {username='oryx', principal='oryx/127.6.29.189@KRBTEST.COM'} at 127.0.0.1:54686
I20260925 04:55:19.450172 13825 tablet_replica.cc:330] T 66ac8ff5456b400fba4189b906d9ded7 P fbde0dd6885c4cd9ab573d65bfc3231e: stopping tablet replica
I20260925 04:55:19.450663 13825 raft_consensus.cc:2229] T 66ac8ff5456b400fba4189b906d9ded7 P fbde0dd6885c4cd9ab573d65bfc3231e [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:55:19.451025 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 13206
I20260925 04:55:19.451174 13826 tablet_replica.cc:330] T 66ac8ff5456b400fba4189b906d9ded7 P 2f7cb902838c4ca382524ae7baa660e4: stopping tablet replica
I20260925 04:55:19.451287 13825 raft_consensus.cc:2258] T 66ac8ff5456b400fba4189b906d9ded7 P fbde0dd6885c4cd9ab573d65bfc3231e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:55:19.451962 13826 raft_consensus.cc:2229] T 66ac8ff5456b400fba4189b906d9ded7 P 2f7cb902838c4ca382524ae7baa660e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:55:19.452180 13827 tablet_replica.cc:330] T 66ac8ff5456b400fba4189b906d9ded7 P dad67e0d8c3a4ab8adeda07f8db9a9a6: stopping tablet replica
I20260925 04:55:19.452279 13825 ts_tablet_manager.cc:1905] T 66ac8ff5456b400fba4189b906d9ded7 P fbde0dd6885c4cd9ab573d65bfc3231e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:55:19.452385 13827 raft_consensus.cc:2229] T 66ac8ff5456b400fba4189b906d9ded7 P dad67e0d8c3a4ab8adeda07f8db9a9a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:55:19.452698 13827 raft_consensus.cc:2258] T 66ac8ff5456b400fba4189b906d9ded7 P dad67e0d8c3a4ab8adeda07f8db9a9a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:55:19.452719 13826 raft_consensus.cc:2258] T 66ac8ff5456b400fba4189b906d9ded7 P 2f7cb902838c4ca382524ae7baa660e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:55:19.453989 13827 ts_tablet_manager.cc:1905] T 66ac8ff5456b400fba4189b906d9ded7 P dad67e0d8c3a4ab8adeda07f8db9a9a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:55:19.454718 13826 ts_tablet_manager.cc:1905] T 66ac8ff5456b400fba4189b906d9ded7 P 2f7cb902838c4ca382524ae7baa660e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:55:19.459689 13825 ts_tablet_manager.cc:1918] T 66ac8ff5456b400fba4189b906d9ded7 P fbde0dd6885c4cd9ab573d65bfc3231e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:55:19.459859 13825 log.cc:1196] T 66ac8ff5456b400fba4189b906d9ded7 P fbde0dd6885c4cd9ab573d65bfc3231e: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-2/wal/wals/66ac8ff5456b400fba4189b906d9ded7
I20260925 04:55:19.460296 13825 ts_tablet_manager.cc:1939] T 66ac8ff5456b400fba4189b906d9ded7 P fbde0dd6885c4cd9ab573d65bfc3231e: Deleting consensus metadata
I20260925 04:55:19.460721 13827 ts_tablet_manager.cc:1918] T 66ac8ff5456b400fba4189b906d9ded7 P dad67e0d8c3a4ab8adeda07f8db9a9a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:55:19.460901 13827 log.cc:1196] T 66ac8ff5456b400fba4189b906d9ded7 P dad67e0d8c3a4ab8adeda07f8db9a9a6: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-0/wal/wals/66ac8ff5456b400fba4189b906d9ded7
I20260925 04:55:19.461417 13827 ts_tablet_manager.cc:1939] T 66ac8ff5456b400fba4189b906d9ded7 P dad67e0d8c3a4ab8adeda07f8db9a9a6: Deleting consensus metadata
I20260925 04:55:19.461722 13019 catalog_manager.cc:4841] TS fbde0dd6885c4cd9ab573d65bfc3231e (127.6.29.131:44161): tablet 66ac8ff5456b400fba4189b906d9ded7 (table test-table [id=523712709bbb4a309922220aaa288a41]) successfully deleted
I20260925 04:55:19.462633 13018 catalog_manager.cc:4841] TS dad67e0d8c3a4ab8adeda07f8db9a9a6 (127.6.29.129:46555): tablet 66ac8ff5456b400fba4189b906d9ded7 (table test-table [id=523712709bbb4a309922220aaa288a41]) successfully deleted
I20260925 04:55:19.463850 13826 ts_tablet_manager.cc:1918] T 66ac8ff5456b400fba4189b906d9ded7 P 2f7cb902838c4ca382524ae7baa660e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:55:19.464023 13826 log.cc:1196] T 66ac8ff5456b400fba4189b906d9ded7 P 2f7cb902838c4ca382524ae7baa660e4: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790312053666503-6262-0/minicluster-data/ts-1/wal/wals/66ac8ff5456b400fba4189b906d9ded7
I20260925 04:55:19.464502 13826 ts_tablet_manager.cc:1939] T 66ac8ff5456b400fba4189b906d9ded7 P 2f7cb902838c4ca382524ae7baa660e4: Deleting consensus metadata
I20260925 04:55:19.465576 13020 catalog_manager.cc:4841] TS 2f7cb902838c4ca382524ae7baa660e4 (127.6.29.130:45893): tablet 66ac8ff5456b400fba4189b906d9ded7 (table test-table [id=523712709bbb4a309922220aaa288a41]) successfully deleted
I20260925 04:55:19.475683 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:19.453708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56292 (local address 127.6.29.129:46555)
0925 04:55:19.453942 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:19.453951 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:19.461880 (+ 7929us) server_negotiation.cc:407] Connection header received
0925 04:55:19.462084 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:19.462090 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:19.462171 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:19.462252 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:19.463963 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:19.465028 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:19.466398 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:19.466880 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:19.466984 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:19.469525 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:19.469540 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:19.469557 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:19.469590 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:19.472751 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:19.473525 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:19.473533 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:19.473536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:19.473675 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:19.474134 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:19.474144 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:19.474146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:19.474485 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:19.474818 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:19.475056 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 04:55:19.475281 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 04:55:19.476480 13396 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:55:19.571880 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 13206
I20260925 04:55:19.581022 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:19.580464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.0.0.1:42266)
0925 04:55:19.580688 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:19.580834 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 04:55:19.582180 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:19.581769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:42115 (local address 127.0.0.1:42280)
0925 04:55:19.581926 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:19.582025 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:42115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 04:55:19.590723 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 13413
I20260925 04:55:19.604461 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:19.591491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32894 (local address 127.6.29.130:45893)
0925 04:55:19.591696 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:19.591703 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:19.591725 (+ 22us) server_negotiation.cc:407] Connection header received
0925 04:55:19.591895 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:19.591901 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:19.591987 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:19.592087 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:19.593489 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:19.594415 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:19.595648 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:19.596174 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:19.596308 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:19.598720 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:19.598746 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:19.598763 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:19.598796 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:19.601728 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:19.602384 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:19.602394 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:19.602397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:19.602548 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:19.602907 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:19.602912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:19.602916 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:19.603266 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:19.603563 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:19.603882 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 04:55:19.604092 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 04:55:19.605096 13545 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:55:19.699916 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 13413
I20260925 04:55:19.718221 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 13563
I20260925 04:55:19.731928 13798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:19.719045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53530 (local address 127.6.29.131:44161)
0925 04:55:19.719226 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:19.719234 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:19.719297 (+ 63us) server_negotiation.cc:407] Connection header received
0925 04:55:19.719527 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:19.719531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:19.719607 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:19.719728 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:19.721064 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:19.721761 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:19.723025 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:19.723405 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:19.723535 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:19.726126 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:19.726141 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:19.726160 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:19.726193 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:19.728944 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:19.729612 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:19.729622 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:19.729625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:19.729759 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:19.730157 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:19.730164 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:19.730170 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:19.730506 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:19.730824 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:19.731296 (+ 472us) server_negotiation.cc:299] Negotiation successful
0925 04:55:19.731525 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 04:55:19.732645 13686 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:55:19.849915 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 13563
I20260925 04:55:19.871695 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 12890
I20260925 04:55:19.872913 13064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:55:19.994988 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 12890
I20260925 04:55:20.016523 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 12990
I20260925 04:55:20.017676 13253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:55:20.132685 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 12990
2026-09-25T04:55:20Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11932 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:55:20 dist-test-slave-xzjr krb5kdc[13836](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:55:20 dist-test-slave-xzjr krb5kdc[13836](info): set up 2 sockets
Sep 25 04:55:20 dist-test-slave-xzjr krb5kdc[13836](info): commencing operation
krb5kdc: starting...
W20260925 04:55:21.241633 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 04:55:21 dist-test-slave-xzjr krb5kdc[13836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:55:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:55:21Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.29.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.29.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.29.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.190.keytab.
Entry for principal oryx/127.6.29.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.190.keytab.
I20260925 04:55:21.461843 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:34931
--webserver_interface=127.6.29.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.190.keytab
--principal=oryx/127.6.29.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:41659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:34931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:55:21.794723 13849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:55:21.795051 13849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:55:21.795142 13849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:55:21.802600 13849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:55:21.802711 13849 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 04:55:21.802749 13849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:55:21.802804 13849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:55:21.802853 13849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:55:21.802913 13849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:55:21.812849 13849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:41659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:34931
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.190.keytab
--principal=oryx/127.6.29.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:34931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:55:21.814129 13849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:55:21.815397 13849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:55:21.827890 13855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:21.827847 13857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:21.827935 13854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:55:21.827867 13849 server_base.cc:1034] running on GCE node
I20260925 04:55:21.829267 13849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:55:21.830648 13849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:55:21.831941 13849 hybrid_clock.cc:648] HybridClock initialized: now 1790312121831866 us; error 31 us; skew 500 ppm
I20260925 04:55:21.832285 13849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:55:21 dist-test-slave-xzjr krb5kdc[13836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312121, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.29.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:55:21.835422 13849 init.cc:377] Logged in from keytab as oryx/127.6.29.190@KRBTEST.COM (short username oryx)
I20260925 04:55:21.836939 13849 webserver.cc:460] Webserver started at http://127.6.29.190:40405/ using document root <none> and password file <none>
I20260925 04:55:21.837628 13849 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:21.837719 13849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:21.837990 13849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:55:21.841033 13849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "94caa156203345ea9f4f18a77e0f320f"
format_stamp: "Formatted at 2026-09-25 04:55:21 on dist-test-slave-xzjr"
server_key: "e3d431db03381bdf19ac67ed4450bf07"
server_key_iv: "cd37e8312ece50ac52b691dad78e4189"
server_key_version: "encryptionkey@0"
I20260925 04:55:21.841807 13849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "94caa156203345ea9f4f18a77e0f320f"
format_stamp: "Formatted at 2026-09-25 04:55:21 on dist-test-slave-xzjr"
server_key: "e3d431db03381bdf19ac67ed4450bf07"
server_key_iv: "cd37e8312ece50ac52b691dad78e4189"
server_key_version: "encryptionkey@0"
I20260925 04:55:21.847349 13849 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 04:55:21.850981 13864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:21.852432 13849 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 04:55:21.852579 13849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "94caa156203345ea9f4f18a77e0f320f"
format_stamp: "Formatted at 2026-09-25 04:55:21 on dist-test-slave-xzjr"
server_key: "e3d431db03381bdf19ac67ed4450bf07"
server_key_iv: "cd37e8312ece50ac52b691dad78e4189"
server_key_version: "encryptionkey@0"
I20260925 04:55:21.852777 13849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:21.882856 13849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:55:21.887277 13849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:55:21.887506 13849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:55:21.909431 13849 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:34931
I20260925 04:55:21.909454 13925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:34931 every 8 connection(s)
I20260925 04:55:21.910822 13849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:55:21.914479 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 13849
I20260925 04:55:21.914709 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 04:55:21.915372 13926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:55:21.915503 6262 external_mini_cluster.cc:1428] Setting key c9fe1bf1291231f533864dc76e7a952d
I20260925 04:55:21.927718 13926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94caa156203345ea9f4f18a77e0f320f: Bootstrap starting.
Sep 25 04:55:21 dist-test-slave-xzjr krb5kdc[13836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM
I20260925 04:55:21.931046 13926 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94caa156203345ea9f4f18a77e0f320f: Neither blocks nor log segments found. Creating new log.
I20260925 04:55:21.932344 13926 log.cc:824] T 00000000000000000000000000000000 P 94caa156203345ea9f4f18a77e0f320f: Log is configured to *not* fsync() on all Append() calls
I20260925 04:55:21.935204 13926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94caa156203345ea9f4f18a77e0f320f: No bootstrap required, opened a new log
I20260925 04:55:21.938046 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:21.917614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37206 (local address 127.6.29.190:34931)
0925 04:55:21.918343 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:21.918355 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:21.918404 (+ 49us) server_negotiation.cc:407] Connection header received
0925 04:55:21.920325 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:21.920347 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:21.920732 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:21.921125 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:21.922510 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:21.923926 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:21.925224 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:21.925948 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:21.926151 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:21.930068 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:21.930076 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:21.930091 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:21.930134 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:21.933453 (+ 3319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:21.934145 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:21.934153 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:21.934159 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:21.934333 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:21.934753 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:21.934762 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:21.934764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:21.935286 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:21.935557 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:21.936034 (+ 477us) server_negotiation.cc:299] Negotiation successful
0925 04:55:21.936790 (+ 756us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":331,"threads_started":1}
I20260925 04:55:21.944070 13926 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94caa156203345ea9f4f18a77e0f320f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94caa156203345ea9f4f18a77e0f320f" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 34931 } }
I20260925 04:55:21.944463 13926 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94caa156203345ea9f4f18a77e0f320f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:55:21.944582 13926 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94caa156203345ea9f4f18a77e0f320f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94caa156203345ea9f4f18a77e0f320f, State: Initialized, Role: FOLLOWER
I20260925 04:55:21.945205 13926 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94caa156203345ea9f4f18a77e0f320f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94caa156203345ea9f4f18a77e0f320f" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 34931 } }
I20260925 04:55:21.945493 13926 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94caa156203345ea9f4f18a77e0f320f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:55:21.945659 13926 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94caa156203345ea9f4f18a77e0f320f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:55:21.945858 13926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94caa156203345ea9f4f18a77e0f320f [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:55:21.948143 13926 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94caa156203345ea9f4f18a77e0f320f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94caa156203345ea9f4f18a77e0f320f" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 34931 } }
I20260925 04:55:21.948688 13926 leader_election.cc:302] T 00000000000000000000000000000000 P 94caa156203345ea9f4f18a77e0f320f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94caa156203345ea9f4f18a77e0f320f; no voters:
I20260925 04:55:21.949362 13926 leader_election.cc:288] T 00000000000000000000000000000000 P 94caa156203345ea9f4f18a77e0f320f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:55:21.949520 13931 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94caa156203345ea9f4f18a77e0f320f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:55:21.950666 13931 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94caa156203345ea9f4f18a77e0f320f [term 1 LEADER]: Becoming Leader. State: Replica: 94caa156203345ea9f4f18a77e0f320f, State: Running, Role: LEADER
I20260925 04:55:21.951256 13931 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94caa156203345ea9f4f18a77e0f320f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94caa156203345ea9f4f18a77e0f320f" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 34931 } }
I20260925 04:55:21.951675 13926 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94caa156203345ea9f4f18a77e0f320f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:55:21.954164 13932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94caa156203345ea9f4f18a77e0f320f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94caa156203345ea9f4f18a77e0f320f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94caa156203345ea9f4f18a77e0f320f" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 34931 } } }
I20260925 04:55:21.954608 13932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94caa156203345ea9f4f18a77e0f320f [sys.catalog]: This master's current role is: LEADER
I20260925 04:55:21.954221 13933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94caa156203345ea9f4f18a77e0f320f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94caa156203345ea9f4f18a77e0f320f. Latest consensus state: current_term: 1 leader_uuid: "94caa156203345ea9f4f18a77e0f320f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94caa156203345ea9f4f18a77e0f320f" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 34931 } } }
I20260925 04:55:21.954859 13933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94caa156203345ea9f4f18a77e0f320f [sys.catalog]: This master's current role is: LEADER
I20260925 04:55:21.956684 13940 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:55:21.961588 13940 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:55:21.970223 13940 catalog_manager.cc:1347] Generated new cluster ID: 1d3826de95fa435f95951f8a963bff32
I20260925 04:55:21.970340 13940 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:55:21.991318 13940 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:55:21.993234 13940 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:55:22.002141 13940 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94caa156203345ea9f4f18a77e0f320f: Generated new TSK 0
I20260925 04:55:22.003024 13940 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.29.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.29.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.29.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.129.keytab.
Entry for principal oryx/127.6.29.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.129.keytab.
I20260925 04:55:22.058036 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.129:0
--local_ip_for_outbound_sockets=127.6.29.129
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:34931
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.129.keytab
--principal=oryx/127.6.29.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:41659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:55:22.393965 13952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:55:22.394315 13952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:55:22.394498 13952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:55:22.401758 13952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:55:22.401947 13952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:55:22.402122 13952 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 04:55:22.402271 13952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.129
I20260925 04:55:22.412115 13952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:41659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.129.keytab
--principal=oryx/127.6.29.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.29.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.29.190:34931
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.129
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:55:22.413477 13952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:55:22.414928 13952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:55:22.428110 13958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:22.428614 13957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:22.428608 13960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:55:22.429167 13952 server_base.cc:1034] running on GCE node
I20260925 04:55:22.429818 13952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:55:22.430930 13952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:55:22.432171 13952 hybrid_clock.cc:648] HybridClock initialized: now 1790312122432113 us; error 47 us; skew 500 ppm
I20260925 04:55:22.432479 13952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:55:22 dist-test-slave-xzjr krb5kdc[13836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312122, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.29.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:55:22.435559 13952 init.cc:377] Logged in from keytab as oryx/127.6.29.129@KRBTEST.COM (short username oryx)
I20260925 04:55:22.437135 13952 webserver.cc:460] Webserver started at http://127.6.29.129:39379/ using document root <none> and password file <none>
I20260925 04:55:22.437819 13952 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:22.437903 13952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:22.438179 13952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:55:22.441285 13952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-0/data/instance:
uuid: "0f15e3c842c043d6938d92d91a857968"
format_stamp: "Formatted at 2026-09-25 04:55:22 on dist-test-slave-xzjr"
server_key: "b70f1802a0558f262fc83722f803bd8b"
server_key_iv: "433e454d15cf6d930348d4c4a1238698"
server_key_version: "encryptionkey@0"
I20260925 04:55:22.442073 13952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance:
uuid: "0f15e3c842c043d6938d92d91a857968"
format_stamp: "Formatted at 2026-09-25 04:55:22 on dist-test-slave-xzjr"
server_key: "b70f1802a0558f262fc83722f803bd8b"
server_key_iv: "433e454d15cf6d930348d4c4a1238698"
server_key_version: "encryptionkey@0"
I20260925 04:55:22.447273 13952 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260925 04:55:22.450747 13967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:22.452250 13952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:55:22.452423 13952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-0/wal
uuid: "0f15e3c842c043d6938d92d91a857968"
format_stamp: "Formatted at 2026-09-25 04:55:22 on dist-test-slave-xzjr"
server_key: "b70f1802a0558f262fc83722f803bd8b"
server_key_iv: "433e454d15cf6d930348d4c4a1238698"
server_key_version: "encryptionkey@0"
I20260925 04:55:22.452637 13952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:22.476177 13952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:55:22.480408 13952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:55:22.480690 13952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:55:22.483532 13952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:55:22.483639 13952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:22.483803 13952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:55:22.483904 13952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Sep 25 04:55:22 dist-test-slave-xzjr krb5kdc[13836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312122, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.29.129@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM
I20260925 04:55:22.507319 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:22.484108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:56481 (local address 127.6.29.190:34931)
0925 04:55:22.484991 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:22.484996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:22.485924 (+ 928us) server_negotiation.cc:407] Connection header received
0925 04:55:22.488040 (+ 2116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:22.488047 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:22.488144 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:22.488267 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:22.490747 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:22.492113 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:22.494656 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:22.495315 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:22.495457 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:22.501122 (+ 5665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:22.501127 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:22.501131 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:22.501179 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:22.503919 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:22.504890 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:22.504897 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:22.504904 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:22.505047 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:22.505509 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:22.505517 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:22.505520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:22.505807 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:22.505967 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:22.506707 (+ 740us) server_negotiation.cc:299] Negotiation successful
0925 04:55:22.506937 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":774,"thread_start_us":687,"threads_started":1}
I20260925 04:55:22.508860 13977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:22.484864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:34931 (local address 127.6.29.129:56481)
0925 04:55:22.485751 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:22.485798 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:22.487669 (+ 1871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:22.488598 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:22.488615 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:22.489548 (+ 933us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:22.490472 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:22.490519 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:22.492342 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:22.492349 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:22.494500 (+ 2151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:22.494513 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:22.495593 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:22.495601 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:22.495953 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:22.496958 (+ 1005us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:22.497004 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:22.500839 (+ 3835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:22.504097 (+ 3258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:22.504108 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:22.504134 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:22.504746 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:22.505166 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:22.505171 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:22.505175 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:22.505372 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:22.505942 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:22.505951 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:22.506345 (+ 394us) client_negotiation.cc:769] Sending connection context
0925 04:55:22.506635 (+ 290us) client_negotiation.cc:240] Negotiation successful
0925 04:55:22.507193 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":394,"threads_started":1}
I20260925 04:55:22.534547 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:22.515667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:34183 (local address 127.6.29.190:34931)
0925 04:55:22.515890 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:22.515896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:22.516967 (+ 1071us) server_negotiation.cc:407] Connection header received
0925 04:55:22.517284 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:22.517292 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:22.517392 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:22.517503 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:22.519350 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:22.520374 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:22.523270 (+ 2896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:22.524024 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:22.524193 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:22.527151 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:22.527156 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:22.527159 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:22.527192 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:22.530785 (+ 3593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:22.532055 (+ 1270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:22.532062 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:22.532075 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:22.532252 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:22.532784 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:22.532791 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:22.532796 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:22.533203 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:22.533362 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:22.534093 (+ 731us) server_negotiation.cc:299] Negotiation successful
0925 04:55:22.534311 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 04:55:22.534821 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:22.515792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:34931 (local address 127.6.29.129:34183)
0925 04:55:22.516835 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:22.516858 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:22.517126 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:22.517593 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:22.517603 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:22.518493 (+ 890us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:22.519136 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:22.519158 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:22.520593 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:22.520599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:22.522928 (+ 2329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:22.522949 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:22.524283 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:22.524296 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:22.524567 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:22.525547 (+ 980us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:22.525583 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:22.526938 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:22.530936 (+ 3998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:22.530944 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:22.530951 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:22.531900 (+ 949us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:22.532400 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:22.532411 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:22.532418 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:22.532629 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:22.533368 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:22.533391 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:22.533746 (+ 355us) client_negotiation.cc:769] Sending connection context
0925 04:55:22.533974 (+ 228us) client_negotiation.cc:240] Negotiation successful
0925 04:55:22.534351 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"thread_start_us":770,"threads_started":1}
I20260925 04:55:22.536405 13952 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.129:39065
I20260925 04:55:22.536414 14087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.129:39065 every 8 connection(s)
I20260925 04:55:22.538012 13952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
I20260925 04:55:22.540232 14088 heartbeater.cc:347] Connected to a master server at 127.6.29.190:34931
I20260925 04:55:22.540475 14088 heartbeater.cc:464] Registering TS with master...
I20260925 04:55:22.540544 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 13952
I20260925 04:55:22.540761 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance
I20260925 04:55:22.541314 14088 heartbeater.cc:511] Master 127.6.29.190:34931 requested a full tablet report, sending...
I20260925 04:55:22.541654 6262 external_mini_cluster.cc:1428] Setting key 9d2532288a7fa50c05e21d08d22997a1
I20260925 04:55:22.543886 13880 ts_manager.cc:194] Registered new tserver with Master: 0f15e3c842c043d6938d92d91a857968 (127.6.29.129:39065)
I20260925 04:55:22.545627 13880 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.29.129@KRBTEST.COM'} at 127.6.29.129:56481
WARNING: no policy specified for oryx/127.6.29.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.29.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.29.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.130.keytab.
Entry for principal oryx/127.6.29.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.130.keytab.
I20260925 04:55:22.595996 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.130:0
--local_ip_for_outbound_sockets=127.6.29.130
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:34931
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.130.keytab
--principal=oryx/127.6.29.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:41659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:55:22.934939 14092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:55:22.935227 14092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:55:22.935324 14092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:55:22.942710 14092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:55:22.942823 14092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:55:22.942909 14092 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 04:55:22.942989 14092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.130
I20260925 04:55:22.953099 14092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:41659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.130.keytab
--principal=oryx/127.6.29.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.29.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.29.190:34931
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.130
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:55:22.954572 14092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:55:22.955986 14092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:55:22.970109 14097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:22.969981 14100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:22.970003 14098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:55:22.971604 14092 server_base.cc:1034] running on GCE node
I20260925 04:55:22.972275 14092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:55:22.973193 14092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:55:22.974407 14092 hybrid_clock.cc:648] HybridClock initialized: now 1790312122974352 us; error 36 us; skew 500 ppm
I20260925 04:55:22.974716 14092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:55:22 dist-test-slave-xzjr krb5kdc[13836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312122, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.29.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:55:22.977921 14092 init.cc:377] Logged in from keytab as oryx/127.6.29.130@KRBTEST.COM (short username oryx)
I20260925 04:55:22.979334 14092 webserver.cc:460] Webserver started at http://127.6.29.130:43865/ using document root <none> and password file <none>
I20260925 04:55:22.980126 14092 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:22.980222 14092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:22.980571 14092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:55:22.983412 14092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-1/data/instance:
uuid: "63173e501f9d48e5953533d6e2a73cff"
format_stamp: "Formatted at 2026-09-25 04:55:22 on dist-test-slave-xzjr"
server_key: "857d0ba0fe2d2689a6d839a7159054cc"
server_key_iv: "719e1bc060dbd2c2a2d5f2fbdc9441b5"
server_key_version: "encryptionkey@0"
I20260925 04:55:22.984233 14092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance:
uuid: "63173e501f9d48e5953533d6e2a73cff"
format_stamp: "Formatted at 2026-09-25 04:55:22 on dist-test-slave-xzjr"
server_key: "857d0ba0fe2d2689a6d839a7159054cc"
server_key_iv: "719e1bc060dbd2c2a2d5f2fbdc9441b5"
server_key_version: "encryptionkey@0"
I20260925 04:55:22.989282 14092 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260925 04:55:22.993106 14107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:22.994594 14092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 04:55:22.994786 14092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-1/wal
uuid: "63173e501f9d48e5953533d6e2a73cff"
format_stamp: "Formatted at 2026-09-25 04:55:22 on dist-test-slave-xzjr"
server_key: "857d0ba0fe2d2689a6d839a7159054cc"
server_key_iv: "719e1bc060dbd2c2a2d5f2fbdc9441b5"
server_key_version: "encryptionkey@0"
I20260925 04:55:22.995069 14092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:23.010242 14092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:55:23.014782 14092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:55:23.015072 14092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:55:23.017696 14092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:55:23.017808 14092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:23.018023 14092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:55:23.018102 14092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 04:55:23 dist-test-slave-xzjr krb5kdc[13836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312122, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.29.130@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM
I20260925 04:55:23.049518 14116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:23.019567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:34931 (local address 127.6.29.130:57601)
0925 04:55:23.020431 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:23.020466 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:23.022355 (+ 1889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:23.023182 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:23.023191 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:23.024068 (+ 877us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:23.024961 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:23.024992 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:23.026740 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:23.026749 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:23.029012 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:23.029023 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:23.030318 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:23.030328 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:23.030527 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:23.035560 (+ 5033us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:23.035642 (+ 82us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:23.039328 (+ 3686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:23.043582 (+ 4254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:23.043593 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:23.043622 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:23.044325 (+ 703us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:23.044910 (+ 585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:23.044919 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:23.044925 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:23.045279 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:23.046134 (+ 855us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:23.046146 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:23.046596 (+ 450us) client_negotiation.cc:769] Sending connection context
0925 04:55:23.047050 (+ 454us) client_negotiation.cc:240] Negotiation successful
0925 04:55:23.047701 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":401,"threads_started":1}
I20260925 04:55:23.049759 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:23.018664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:57601 (local address 127.6.29.190:34931)
0925 04:55:23.018856 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:23.018863 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:23.020586 (+ 1723us) server_negotiation.cc:407] Connection header received
0925 04:55:23.022735 (+ 2149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:23.022742 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:23.022837 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:23.022956 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:23.025228 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:23.026550 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:23.029167 (+ 2617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:23.029986 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:23.030157 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:23.039706 (+ 9549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:23.039714 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:23.039719 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:23.039786 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:23.043415 (+ 3629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:23.044513 (+ 1098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:23.044522 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:23.044528 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:23.044730 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:23.045475 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:23.045484 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:23.045490 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:23.045956 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:23.046160 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:23.048042 (+ 1882us) server_negotiation.cc:299] Negotiation successful
0925 04:55:23.048282 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 04:55:23.066349 14092 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.130:45827
I20260925 04:55:23.066322 14226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.130:45827 every 8 connection(s)
I20260925 04:55:23.067740 14092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
I20260925 04:55:23.068598 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 14092
I20260925 04:55:23.068854 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance
I20260925 04:55:23.069500 6262 external_mini_cluster.cc:1428] Setting key af57218ad4070ca38cf2138d3fba7ee6
I20260925 04:55:23.070307 14214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:23.056522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:34931 (local address 127.6.29.130:41735)
0925 04:55:23.057381 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:23.057421 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:23.057592 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:23.057919 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:23.057924 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:23.058391 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:23.058714 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:23.058725 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:23.059702 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:23.059712 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:23.061644 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:23.061657 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:23.062446 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:23.062452 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:23.062637 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:23.063300 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:23.063317 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:23.064262 (+ 945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:23.067571 (+ 3309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:23.067577 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:23.067581 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:23.068171 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:23.068536 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:23.068541 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:23.068545 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:23.068678 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:23.069316 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:23.069321 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:23.069495 (+ 174us) client_negotiation.cc:769] Sending connection context
0925 04:55:23.069637 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 04:55:23.069889 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"thread_start_us":497,"threads_started":1}
I20260925 04:55:23.070906 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:23.056402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:41735 (local address 127.6.29.190:34931)
0925 04:55:23.056576 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:23.056582 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:23.057492 (+ 910us) server_negotiation.cc:407] Connection header received
0925 04:55:23.057710 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:23.057715 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:23.057791 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:23.057894 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:23.058860 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:23.059560 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:23.061790 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:23.062254 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:23.062403 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:23.064451 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:23.064459 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:23.064468 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:23.064499 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:23.067440 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:23.068287 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:23.068294 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:23.068297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:23.068429 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:23.068829 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:23.068835 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:23.068844 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:23.069200 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:23.069352 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:23.070390 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0925 04:55:23.070601 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 04:55:23.079505 14227 heartbeater.cc:347] Connected to a master server at 127.6.29.190:34931
I20260925 04:55:23.079808 14227 heartbeater.cc:464] Registering TS with master...
I20260925 04:55:23.080741 14227 heartbeater.cc:511] Master 127.6.29.190:34931 requested a full tablet report, sending...
I20260925 04:55:23.082240 13881 ts_manager.cc:194] Registered new tserver with Master: 63173e501f9d48e5953533d6e2a73cff (127.6.29.130:45827)
I20260925 04:55:23.083096 13881 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.29.130@KRBTEST.COM'} at 127.6.29.130:57601
WARNING: no policy specified for oryx/127.6.29.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.29.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.29.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.131.keytab.
Entry for principal oryx/127.6.29.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.131.keytab.
I20260925 04:55:23.121025 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.131:0
--local_ip_for_outbound_sockets=127.6.29.131
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:34931
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.131.keytab
--principal=oryx/127.6.29.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:41659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:55:23.449077 14231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:55:23.449419 14231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:55:23.449590 14231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:55:23.457406 14231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:55:23.457556 14231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:55:23.457707 14231 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 04:55:23.457844 14231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.131
I20260925 04:55:23.467823 14231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:41659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.131.keytab
--principal=oryx/127.6.29.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.29.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.29.190:34931
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.131
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:55:23.469244 14231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:55:23.470597 14231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:55:23.484401 14236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:23.484401 14237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:23.484428 14239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:55:23.485325 14231 server_base.cc:1034] running on GCE node
I20260925 04:55:23.485936 14231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:55:23.486871 14231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:55:23.488085 14231 hybrid_clock.cc:648] HybridClock initialized: now 1790312123488019 us; error 35 us; skew 500 ppm
I20260925 04:55:23.488394 14231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:55:23 dist-test-slave-xzjr krb5kdc[13836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312123, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.29.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:55:23.491580 14231 init.cc:377] Logged in from keytab as oryx/127.6.29.131@KRBTEST.COM (short username oryx)
I20260925 04:55:23.493135 14231 webserver.cc:460] Webserver started at http://127.6.29.131:45569/ using document root <none> and password file <none>
I20260925 04:55:23.493844 14231 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:23.493974 14231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:23.494294 14231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:55:23.497282 14231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-2/data/instance:
uuid: "68fc8818ace94823866a0260adc36dcb"
format_stamp: "Formatted at 2026-09-25 04:55:23 on dist-test-slave-xzjr"
server_key: "bddf1813430590486e1bcad1c923a6c1"
server_key_iv: "a7607b204fef75805812c4601237e7d0"
server_key_version: "encryptionkey@0"
I20260925 04:55:23.498111 14231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance:
uuid: "68fc8818ace94823866a0260adc36dcb"
format_stamp: "Formatted at 2026-09-25 04:55:23 on dist-test-slave-xzjr"
server_key: "bddf1813430590486e1bcad1c923a6c1"
server_key_iv: "a7607b204fef75805812c4601237e7d0"
server_key_version: "encryptionkey@0"
I20260925 04:55:23.503480 14231 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 04:55:23.507097 14246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:23.508632 14231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:55:23.508816 14231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-2/wal
uuid: "68fc8818ace94823866a0260adc36dcb"
format_stamp: "Formatted at 2026-09-25 04:55:23 on dist-test-slave-xzjr"
server_key: "bddf1813430590486e1bcad1c923a6c1"
server_key_iv: "a7607b204fef75805812c4601237e7d0"
server_key_version: "encryptionkey@0"
I20260925 04:55:23.509039 14231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:23.528432 14231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:55:23.532979 14231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:55:23.533257 14231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:55:23.535988 14231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:55:23.536118 14231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:23.536267 14231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:55:23.536345 14231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:23.549400 14088 heartbeater.cc:503] Master 127.6.29.190:34931 was elected leader, sending a full tablet report...
Sep 25 04:55:23 dist-test-slave-xzjr krb5kdc[13836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312123, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.29.131@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM
I20260925 04:55:23.563105 14255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:23.537346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:34931 (local address 127.6.29.131:33083)
0925 04:55:23.538298 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:23.538347 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:23.541329 (+ 2982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:23.542437 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:23.542454 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:23.543260 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:23.544279 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:23.544311 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:23.545731 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:23.545740 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:23.548628 (+ 2888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:23.548651 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:23.549749 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:23.549759 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:23.550123 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:23.551162 (+ 1039us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:23.551227 (+ 65us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:23.554728 (+ 3501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:23.558344 (+ 3616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:23.558353 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:23.558372 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:23.558816 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:23.559201 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:23.559206 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:23.559209 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:23.559392 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:23.559968 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:23.559978 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:23.560541 (+ 563us) client_negotiation.cc:769] Sending connection context
0925 04:55:23.560895 (+ 354us) client_negotiation.cc:240] Negotiation successful
0925 04:55:23.561431 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":387,"threads_started":1}
I20260925 04:55:23.564263 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:23.536651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:33083 (local address 127.6.29.190:34931)
0925 04:55:23.536873 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:23.536879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:23.538489 (+ 1610us) server_negotiation.cc:407] Connection header received
0925 04:55:23.541687 (+ 3198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:23.541694 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:23.541781 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:23.541853 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:23.544492 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:23.545508 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:23.548854 (+ 3346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:23.549433 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:23.549565 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:23.554959 (+ 5394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:23.554973 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:23.554977 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:23.555030 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:23.558206 (+ 3176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:23.558946 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:23.558952 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:23.558956 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:23.559081 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:23.559511 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:23.559516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:23.559519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:23.559837 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:23.560000 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:23.563650 (+ 3650us) server_negotiation.cc:299] Negotiation successful
0925 04:55:23.563929 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 04:55:23.587074 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:23.570653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:41123 (local address 127.6.29.190:34931)
0925 04:55:23.570799 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:23.570805 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:23.571677 (+ 872us) server_negotiation.cc:407] Connection header received
0925 04:55:23.572039 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:23.572047 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:23.572134 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:23.572209 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:23.573578 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:23.574682 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:23.577450 (+ 2768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:23.577867 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:23.577971 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:23.580900 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:23.580908 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:23.580913 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:23.580959 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:23.584129 (+ 3170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:23.585145 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:23.585151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:23.585156 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:23.585278 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:23.585645 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:23.585650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:23.585660 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:23.585921 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:23.586077 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:23.586486 (+ 409us) server_negotiation.cc:299] Negotiation successful
0925 04:55:23.586708 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 04:55:23.587178 14316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:23.570113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:34931 (local address 127.6.29.131:41123)
0925 04:55:23.571548 (+ 1435us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:23.571570 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:23.571825 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:23.572381 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:23.572384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:23.573024 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:23.573415 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:23.573425 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:23.574863 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:23.574869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:23.577219 (+ 2350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:23.577240 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:23.578085 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:23.578095 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:23.578325 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:23.579087 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:23.579119 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:23.580683 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:23.584333 (+ 3650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:23.584340 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:23.584347 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:23.584961 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:23.585398 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:23.585402 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:23.585406 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:23.585534 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:23.586045 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:23.586050 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:23.586235 (+ 185us) client_negotiation.cc:769] Sending connection context
0925 04:55:23.586368 (+ 133us) client_negotiation.cc:240] Negotiation successful
0925 04:55:23.586714 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1282,"thread_start_us":1072,"threads_started":1}
I20260925 04:55:23.594508 14231 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.131:43831
I20260925 04:55:23.594548 14365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.131:43831 every 8 connection(s)
I20260925 04:55:23.595806 14231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
I20260925 04:55:23.597803 14366 heartbeater.cc:347] Connected to a master server at 127.6.29.190:34931
I20260925 04:55:23.598038 14366 heartbeater.cc:464] Registering TS with master...
I20260925 04:55:23.598728 14366 heartbeater.cc:511] Master 127.6.29.190:34931 requested a full tablet report, sending...
I20260925 04:55:23.600163 13881 ts_manager.cc:194] Registered new tserver with Master: 68fc8818ace94823866a0260adc36dcb (127.6.29.131:43831)
I20260925 04:55:23.600908 13881 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.29.131@KRBTEST.COM'} at 127.6.29.131:33083
I20260925 04:55:23.604770 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 14231
I20260925 04:55:23.604918 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance
I20260925 04:55:23.605270 6262 external_mini_cluster.cc:1428] Setting key 97f53239692fba624431e0fbe3098ceb
I20260925 04:55:23.608058 6262 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 04:55:23 dist-test-slave-xzjr krb5kdc[13836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312123, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 04:55:23 dist-test-slave-xzjr krb5kdc[13836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:23 dist-test-slave-xzjr krb5kdc[13836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
W20260925 04:55:23.656862 13976 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:23.646090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37214 (local address 127.6.29.190:34931)
0925 04:55:23.646314 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:23.646321 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:23.646511 (+ 190us) server_negotiation.cc:407] Connection header received
0925 04:55:23.646762 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:23.646786 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:23.646893 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:23.646992 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:23.648377 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:23.649117 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:23.650355 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:23.650787 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:23.650922 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:23.656520 (+ 5598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37214: BlockingRecv error: recv got EOF from 127.0.0.1:37214 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 25 04:55:23 dist-test-slave-xzjr krb5kdc[13836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312123, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM
I20260925 04:55:23.673487 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:23.660480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37224 (local address 127.6.29.190:34931)
0925 04:55:23.660676 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:23.660681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:23.660882 (+ 201us) server_negotiation.cc:407] Connection header received
0925 04:55:23.661013 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:23.661017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:23.661101 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:23.661186 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:23.662375 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:23.663062 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:23.664495 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:23.664951 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:23.665062 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:23.668890 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:23.668895 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:23.668898 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:23.668934 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:23.671327 (+ 2393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:23.672015 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:23.672019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:23.672021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:23.672131 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:23.672443 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:23.672446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:23.672448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:23.672710 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:23.672889 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:23.673099 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 04:55:23.673257 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 04:55:23.677517 13881 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:55:23.684350 13881 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:55:23.716638 14387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:23.696396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:45827 (local address 127.0.0.1:48120)
0925 04:55:23.707603 (+ 11207us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:23.707657 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:23.707931 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:23.708859 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:23.708867 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:23.708895 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:55:23.709320 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:23.709339 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:23.711189 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:23.711194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:23.713947 (+ 2753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:23.714022 (+ 75us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:23.715709 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:23.715714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:23.715910 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:23.715961 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 04:55:23.716123 (+ 162us) client_negotiation.cc:240] Negotiation successful
0925 04:55:23.716306 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10800,"thread_start_us":13076,"threads_started":1}
I20260925 04:55:23.716634 14385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:23.696582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48120 (local address 127.6.29.130:45827)
0925 04:55:23.703205 (+ 6623us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:23.703219 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:23.707802 (+ 4583us) server_negotiation.cc:407] Connection header received
0925 04:55:23.708233 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:23.708241 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:23.708514 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:23.708723 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:55:23.709571 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:23.711055 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:23.714153 (+ 3098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:23.715540 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:23.715663 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:23.715902 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:23.716154 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 04:55:23.716309 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":709,"threads_started":1}
I20260925 04:55:23.717738 14386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:23.696622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.131:43831 (local address 127.0.0.1:37732)
0925 04:55:23.709863 (+ 13241us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:23.709884 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:23.710012 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:23.710598 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:23.710602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:23.710626 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:55:23.711174 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:23.711186 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:23.713222 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:23.713227 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:23.714959 (+ 1732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:23.714977 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:23.716905 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:23.716911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:23.717086 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:23.717123 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 04:55:23.717227 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 04:55:23.717360 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13108,"mutex_wait_us":1,"thread_start_us":11741,"threads_started":1}
I20260925 04:55:23.718741 14384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:23.697001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:39065 (local address 127.0.0.1:35994)
0925 04:55:23.709976 (+ 12975us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:23.709992 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:23.710091 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:23.710827 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:23.710831 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:23.710853 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:55:23.711460 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:23.711473 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:23.713366 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:23.713372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:23.715478 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:23.715493 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:23.718155 (+ 2662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:23.718161 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:23.718322 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:23.718344 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:55:23.718418 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 04:55:23.718514 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12854,"thread_start_us":11179,"threads_started":1}
I20260925 04:55:23.718930 14389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:23.697547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35994 (local address 127.6.29.129:39065)
0925 04:55:23.709332 (+ 11785us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:23.709341 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:23.710058 (+ 717us) server_negotiation.cc:407] Connection header received
0925 04:55:23.710220 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:23.710227 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:23.710437 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:23.710631 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:55:23.711688 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:23.713224 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:23.715640 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:23.717960 (+ 2320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:23.718108 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:23.718324 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:23.718478 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 04:55:23.718646 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11620,"thread_start_us":11477,"threads_started":1}
I20260925 04:55:23.719552 14388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:23.697237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37732 (local address 127.6.29.131:43831)
0925 04:55:23.709850 (+ 12613us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:23.709857 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:23.709984 (+ 127us) server_negotiation.cc:407] Connection header received
0925 04:55:23.710217 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:23.710223 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:23.710409 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:23.710622 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:55:23.711392 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:23.713091 (+ 1699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:23.715122 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:23.716406 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:23.716549 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:23.716775 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:23.718081 (+ 1306us) server_negotiation.cc:299] Negotiation successful
0925 04:55:23.719326 (+ 1245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12459,"thread_start_us":12366,"threads_started":1}
I20260925 04:55:23.721164 14157 tablet_service.cc:1467] Processing CreateTablet for tablet c990c9b912044467a9d11499fbc303b2 (DEFAULT_TABLE table=test-table [id=ab57c3745d564208907fe9133d16a9d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:55:23.721946 14018 tablet_service.cc:1467] Processing CreateTablet for tablet c990c9b912044467a9d11499fbc303b2 (DEFAULT_TABLE table=test-table [id=ab57c3745d564208907fe9133d16a9d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:55:23.722716 14157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c990c9b912044467a9d11499fbc303b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:55:23.723248 14018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c990c9b912044467a9d11499fbc303b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:55:23.723021 14296 tablet_service.cc:1467] Processing CreateTablet for tablet c990c9b912044467a9d11499fbc303b2 (DEFAULT_TABLE table=test-table [id=ab57c3745d564208907fe9133d16a9d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:55:23.724411 14296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c990c9b912044467a9d11499fbc303b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:55:23.733793 14390 tablet_bootstrap.cc:492] T c990c9b912044467a9d11499fbc303b2 P 0f15e3c842c043d6938d92d91a857968: Bootstrap starting.
I20260925 04:55:23.737025 14390 tablet_bootstrap.cc:654] T c990c9b912044467a9d11499fbc303b2 P 0f15e3c842c043d6938d92d91a857968: Neither blocks nor log segments found. Creating new log.
I20260925 04:55:23.737391 14391 tablet_bootstrap.cc:492] T c990c9b912044467a9d11499fbc303b2 P 63173e501f9d48e5953533d6e2a73cff: Bootstrap starting.
I20260925 04:55:23.739032 14392 tablet_bootstrap.cc:492] T c990c9b912044467a9d11499fbc303b2 P 68fc8818ace94823866a0260adc36dcb: Bootstrap starting.
I20260925 04:55:23.739830 14390 log.cc:824] T c990c9b912044467a9d11499fbc303b2 P 0f15e3c842c043d6938d92d91a857968: Log is configured to *not* fsync() on all Append() calls
I20260925 04:55:23.740903 14391 tablet_bootstrap.cc:654] T c990c9b912044467a9d11499fbc303b2 P 63173e501f9d48e5953533d6e2a73cff: Neither blocks nor log segments found. Creating new log.
I20260925 04:55:23.742159 14391 log.cc:824] T c990c9b912044467a9d11499fbc303b2 P 63173e501f9d48e5953533d6e2a73cff: Log is configured to *not* fsync() on all Append() calls
I20260925 04:55:23.742245 14392 tablet_bootstrap.cc:654] T c990c9b912044467a9d11499fbc303b2 P 68fc8818ace94823866a0260adc36dcb: Neither blocks nor log segments found. Creating new log.
I20260925 04:55:23.743088 14390 tablet_bootstrap.cc:492] T c990c9b912044467a9d11499fbc303b2 P 0f15e3c842c043d6938d92d91a857968: No bootstrap required, opened a new log
I20260925 04:55:23.743369 14390 ts_tablet_manager.cc:1397] T c990c9b912044467a9d11499fbc303b2 P 0f15e3c842c043d6938d92d91a857968: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260925 04:55:23.743571 14392 log.cc:824] T c990c9b912044467a9d11499fbc303b2 P 68fc8818ace94823866a0260adc36dcb: Log is configured to *not* fsync() on all Append() calls
I20260925 04:55:23.745180 14391 tablet_bootstrap.cc:492] T c990c9b912044467a9d11499fbc303b2 P 63173e501f9d48e5953533d6e2a73cff: No bootstrap required, opened a new log
I20260925 04:55:23.745432 14391 ts_tablet_manager.cc:1397] T c990c9b912044467a9d11499fbc303b2 P 63173e501f9d48e5953533d6e2a73cff: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 04:55:23.746078 14392 tablet_bootstrap.cc:492] T c990c9b912044467a9d11499fbc303b2 P 68fc8818ace94823866a0260adc36dcb: No bootstrap required, opened a new log
I20260925 04:55:23.746318 14392 ts_tablet_manager.cc:1397] T c990c9b912044467a9d11499fbc303b2 P 68fc8818ace94823866a0260adc36dcb: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260925 04:55:23.754176 14392 raft_consensus.cc:348] T c990c9b912044467a9d11499fbc303b2 P 68fc8818ace94823866a0260adc36dcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } } peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } }
I20260925 04:55:23.754225 14390 raft_consensus.cc:348] T c990c9b912044467a9d11499fbc303b2 P 0f15e3c842c043d6938d92d91a857968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } } peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } }
I20260925 04:55:23.754607 14392 raft_consensus.cc:374] T c990c9b912044467a9d11499fbc303b2 P 68fc8818ace94823866a0260adc36dcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:55:23.754684 14390 raft_consensus.cc:374] T c990c9b912044467a9d11499fbc303b2 P 0f15e3c842c043d6938d92d91a857968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:55:23.754750 14392 raft_consensus.cc:729] T c990c9b912044467a9d11499fbc303b2 P 68fc8818ace94823866a0260adc36dcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68fc8818ace94823866a0260adc36dcb, State: Initialized, Role: FOLLOWER
I20260925 04:55:23.754832 14390 raft_consensus.cc:729] T c990c9b912044467a9d11499fbc303b2 P 0f15e3c842c043d6938d92d91a857968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f15e3c842c043d6938d92d91a857968, State: Initialized, Role: FOLLOWER
I20260925 04:55:23.755517 14390 consensus_queue.cc:260] T c990c9b912044467a9d11499fbc303b2 P 0f15e3c842c043d6938d92d91a857968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } } peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } }
I20260925 04:55:23.755476 14392 consensus_queue.cc:260] T c990c9b912044467a9d11499fbc303b2 P 68fc8818ace94823866a0260adc36dcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } } peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } }
I20260925 04:55:23.756197 14391 raft_consensus.cc:348] T c990c9b912044467a9d11499fbc303b2 P 63173e501f9d48e5953533d6e2a73cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } } peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } }
I20260925 04:55:23.756776 14391 raft_consensus.cc:374] T c990c9b912044467a9d11499fbc303b2 P 63173e501f9d48e5953533d6e2a73cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:55:23.756902 14391 raft_consensus.cc:729] T c990c9b912044467a9d11499fbc303b2 P 63173e501f9d48e5953533d6e2a73cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63173e501f9d48e5953533d6e2a73cff, State: Initialized, Role: FOLLOWER
I20260925 04:55:23.757655 14391 consensus_queue.cc:260] T c990c9b912044467a9d11499fbc303b2 P 63173e501f9d48e5953533d6e2a73cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } } peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } }
I20260925 04:55:23.758980 14227 heartbeater.cc:503] Master 127.6.29.190:34931 was elected leader, sending a full tablet report...
I20260925 04:55:23.759655 14391 ts_tablet_manager.cc:1428] T c990c9b912044467a9d11499fbc303b2 P 63173e501f9d48e5953533d6e2a73cff: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260925 04:55:23.764071 14366 heartbeater.cc:503] Master 127.6.29.190:34931 was elected leader, sending a full tablet report...
I20260925 04:55:23.765230 14390 ts_tablet_manager.cc:1428] T c990c9b912044467a9d11499fbc303b2 P 0f15e3c842c043d6938d92d91a857968: Time spent starting tablet: real 0.022s user 0.021s sys 0.000s
I20260925 04:55:23.765244 14392 ts_tablet_manager.cc:1428] T c990c9b912044467a9d11499fbc303b2 P 68fc8818ace94823866a0260adc36dcb: Time spent starting tablet: real 0.017s user 0.016s sys 0.001s
W20260925 04:55:23.790717 14089 tablet.cc:2369] T c990c9b912044467a9d11499fbc303b2 P 0f15e3c842c043d6938d92d91a857968: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:55:23.827104 14228 tablet.cc:2369] T c990c9b912044467a9d11499fbc303b2 P 63173e501f9d48e5953533d6e2a73cff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:55:23.833261 14398 raft_consensus.cc:482] T c990c9b912044467a9d11499fbc303b2 P 63173e501f9d48e5953533d6e2a73cff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:55:23.833591 14398 raft_consensus.cc:504] T c990c9b912044467a9d11499fbc303b2 P 63173e501f9d48e5953533d6e2a73cff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } } peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } }
I20260925 04:55:23.835290 14398 leader_election.cc:288] T c990c9b912044467a9d11499fbc303b2 P 63173e501f9d48e5953533d6e2a73cff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68fc8818ace94823866a0260adc36dcb (127.6.29.131:43831), 0f15e3c842c043d6938d92d91a857968 (127.6.29.129:39065)
I20260925 04:55:23.842319 14389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:23.836012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:38859 (local address 127.6.29.129:39065)
0925 04:55:23.836242 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:23.836249 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:23.837025 (+ 776us) server_negotiation.cc:407] Connection header received
0925 04:55:23.837192 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:23.837197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:23.837284 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:23.837385 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:55:23.837961 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:23.838691 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:23.840579 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:23.841313 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:23.841497 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:23.841588 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:23.841935 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 04:55:23.842064 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 04:55:23.842334 14388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:23.835698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:39741 (local address 127.6.29.131:43831)
0925 04:55:23.835954 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:23.835960 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:23.836841 (+ 881us) server_negotiation.cc:407] Connection header received
0925 04:55:23.837014 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:23.837020 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:23.837096 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:23.837183 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:55:23.837767 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:23.838539 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:23.840512 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:23.841317 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:23.841507 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:23.841588 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:23.841936 (+ 348us) server_negotiation.cc:299] Negotiation successful
0925 04:55:23.842068 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260925 04:55:23.842536 14399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:23.835725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.131:43831 (local address 127.6.29.130:39741)
0925 04:55:23.836726 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:23.836742 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:23.836876 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:23.837208 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:23.837213 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:23.837235 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:55:23.837615 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:23.837626 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:23.838668 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:23.838673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:23.840364 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:23.840376 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:23.841574 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:23.841579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:23.841758 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:23.841797 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 04:55:23.841900 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 04:55:23.842055 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":886,"mutex_wait_us":51,"thread_start_us":517,"threads_started":1}
I20260925 04:55:23.842562 14400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:23.835803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:39065 (local address 127.6.29.130:38859)
0925 04:55:23.836885 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:23.836902 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:23.837054 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:23.837411 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:23.837414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:23.837432 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:55:23.837806 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:23.837821 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:23.838834 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:23.838838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:23.840412 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:23.840423 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:23.841562 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:23.841568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:23.841755 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:23.841796 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 04:55:23.841900 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 04:55:23.842062 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":932,"mutex_wait_us":82,"thread_start_us":599,"threads_started":1}
I20260925 04:55:23.843736 14321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c990c9b912044467a9d11499fbc303b2" candidate_uuid: "63173e501f9d48e5953533d6e2a73cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68fc8818ace94823866a0260adc36dcb" is_pre_election: true
I20260925 04:55:23.843801 14043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c990c9b912044467a9d11499fbc303b2" candidate_uuid: "63173e501f9d48e5953533d6e2a73cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f15e3c842c043d6938d92d91a857968" is_pre_election: true
I20260925 04:55:23.844295 14321 raft_consensus.cc:2454] T c990c9b912044467a9d11499fbc303b2 P 68fc8818ace94823866a0260adc36dcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63173e501f9d48e5953533d6e2a73cff in term 0.
I20260925 04:55:23.844295 14043 raft_consensus.cc:2454] T c990c9b912044467a9d11499fbc303b2 P 0f15e3c842c043d6938d92d91a857968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 63173e501f9d48e5953533d6e2a73cff in term 0.
I20260925 04:55:23.844935 14110 leader_election.cc:302] T c990c9b912044467a9d11499fbc303b2 P 63173e501f9d48e5953533d6e2a73cff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0f15e3c842c043d6938d92d91a857968, 63173e501f9d48e5953533d6e2a73cff, 68fc8818ace94823866a0260adc36dcb; no voters:
I20260925 04:55:23.845340 14398 raft_consensus.cc:2781] T c990c9b912044467a9d11499fbc303b2 P 63173e501f9d48e5953533d6e2a73cff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:55:23.845489 14398 raft_consensus.cc:482] T c990c9b912044467a9d11499fbc303b2 P 63173e501f9d48e5953533d6e2a73cff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:55:23.845594 14398 raft_consensus.cc:3037] T c990c9b912044467a9d11499fbc303b2 P 63173e501f9d48e5953533d6e2a73cff [term 0 FOLLOWER]: Advancing to term 1
W20260925 04:55:23.847344 14367 tablet.cc:2369] T c990c9b912044467a9d11499fbc303b2 P 68fc8818ace94823866a0260adc36dcb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:55:23.848006 14398 raft_consensus.cc:504] T c990c9b912044467a9d11499fbc303b2 P 63173e501f9d48e5953533d6e2a73cff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } } peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } }
I20260925 04:55:23.848676 14398 leader_election.cc:288] T c990c9b912044467a9d11499fbc303b2 P 63173e501f9d48e5953533d6e2a73cff [CANDIDATE]: Term 1 election: Requested vote from peers 68fc8818ace94823866a0260adc36dcb (127.6.29.131:43831), 0f15e3c842c043d6938d92d91a857968 (127.6.29.129:39065)
I20260925 04:55:23.849050 14321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c990c9b912044467a9d11499fbc303b2" candidate_uuid: "63173e501f9d48e5953533d6e2a73cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68fc8818ace94823866a0260adc36dcb"
I20260925 04:55:23.849179 14043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c990c9b912044467a9d11499fbc303b2" candidate_uuid: "63173e501f9d48e5953533d6e2a73cff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f15e3c842c043d6938d92d91a857968"
I20260925 04:55:23.849242 14321 raft_consensus.cc:3037] T c990c9b912044467a9d11499fbc303b2 P 68fc8818ace94823866a0260adc36dcb [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:55:23.849378 14043 raft_consensus.cc:3037] T c990c9b912044467a9d11499fbc303b2 P 0f15e3c842c043d6938d92d91a857968 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:55:23.851344 14321 raft_consensus.cc:2454] T c990c9b912044467a9d11499fbc303b2 P 68fc8818ace94823866a0260adc36dcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63173e501f9d48e5953533d6e2a73cff in term 1.
I20260925 04:55:23.851545 14043 raft_consensus.cc:2454] T c990c9b912044467a9d11499fbc303b2 P 0f15e3c842c043d6938d92d91a857968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 63173e501f9d48e5953533d6e2a73cff in term 1.
I20260925 04:55:23.851758 14110 leader_election.cc:302] T c990c9b912044467a9d11499fbc303b2 P 63173e501f9d48e5953533d6e2a73cff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63173e501f9d48e5953533d6e2a73cff, 68fc8818ace94823866a0260adc36dcb; no voters:
I20260925 04:55:23.852141 14398 raft_consensus.cc:2781] T c990c9b912044467a9d11499fbc303b2 P 63173e501f9d48e5953533d6e2a73cff [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:55:23.852588 14398 raft_consensus.cc:686] T c990c9b912044467a9d11499fbc303b2 P 63173e501f9d48e5953533d6e2a73cff [term 1 LEADER]: Becoming Leader. State: Replica: 63173e501f9d48e5953533d6e2a73cff, State: Running, Role: LEADER
I20260925 04:55:23.853128 14398 consensus_queue.cc:237] T c990c9b912044467a9d11499fbc303b2 P 63173e501f9d48e5953533d6e2a73cff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } } peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } }
I20260925 04:55:23.858063 13879 catalog_manager.cc:5495] T c990c9b912044467a9d11499fbc303b2 P 63173e501f9d48e5953533d6e2a73cff reported cstate change: term changed from 0 to 1, leader changed from <none> to 63173e501f9d48e5953533d6e2a73cff (127.6.29.130). New cstate: current_term: 1 leader_uuid: "63173e501f9d48e5953533d6e2a73cff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } health_report { overall_health: UNKNOWN } } }
I20260925 04:55:23.902908 14385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:23.898028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48132 (local address 127.6.29.130:45827)
0925 04:55:23.898255 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:23.898259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:23.898277 (+ 18us) server_negotiation.cc:407] Connection header received
0925 04:55:23.898468 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:23.898471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:23.898556 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:23.898690 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:55:23.898996 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:23.899829 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:23.901080 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:23.901468 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:23.901565 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:23.901894 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:55:23.902454 (+ 560us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:55:23.902558 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:23.902633 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 04:55:23.902693 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260925 04:55:23.911275 14043 raft_consensus.cc:1261] T c990c9b912044467a9d11499fbc303b2 P 0f15e3c842c043d6938d92d91a857968 [term 1 FOLLOWER]: Refusing update from remote peer 63173e501f9d48e5953533d6e2a73cff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:55:23.911275 14321 raft_consensus.cc:1261] T c990c9b912044467a9d11499fbc303b2 P 68fc8818ace94823866a0260adc36dcb [term 1 FOLLOWER]: Refusing update from remote peer 63173e501f9d48e5953533d6e2a73cff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:55:23.912271 14401 consensus_queue.cc:1035] T c990c9b912044467a9d11499fbc303b2 P 63173e501f9d48e5953533d6e2a73cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:55:23.912765 14398 consensus_queue.cc:1035] T c990c9b912044467a9d11499fbc303b2 P 63173e501f9d48e5953533d6e2a73cff [LEADER]: Connected to new peer: Peer: permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:55:23.933414 14406 mvcc.cc:205] Tried to move back new op lower bound from 7333118459530268672 to 7333118459306778624. Current Snapshot: MvccSnapshot[applied={T|T < 7333118459530268672}]
I20260925 04:55:23.953557 13881 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:55:23.953878 13881 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:55:23.957700 13881 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 94caa156203345ea9f4f18a77e0f320f: Sending DeleteTablet for 3 replicas of tablet c990c9b912044467a9d11499fbc303b2
I20260925 04:55:23.958859 14157 tablet_service.cc:1514] Processing DeleteTablet for tablet c990c9b912044467a9d11499fbc303b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:55:23 UTC) from {username='oryx', principal='oryx/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:48120
I20260925 04:55:23.959722 14296 tablet_service.cc:1514] Processing DeleteTablet for tablet c990c9b912044467a9d11499fbc303b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:55:23 UTC) from {username='oryx', principal='oryx/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:37732
I20260925 04:55:23.959875 14413 tablet_replica.cc:330] T c990c9b912044467a9d11499fbc303b2 P 63173e501f9d48e5953533d6e2a73cff: stopping tablet replica
I20260925 04:55:23.960034 14018 tablet_service.cc:1514] Processing DeleteTablet for tablet c990c9b912044467a9d11499fbc303b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:55:23 UTC) from {username='oryx', principal='oryx/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:35994
I20260925 04:55:23.960564 14413 raft_consensus.cc:2229] T c990c9b912044467a9d11499fbc303b2 P 63173e501f9d48e5953533d6e2a73cff [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:55:23.960810 13881 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37224:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:55:23.961467 13881 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:55:23.961450 14413 raft_consensus.cc:2258] T c990c9b912044467a9d11499fbc303b2 P 63173e501f9d48e5953533d6e2a73cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:55:23.968706 14017 tablet_service.cc:1467] Processing CreateTablet for tablet 5f1e731a431940168737a5e6b789f29c (DEFAULT_TABLE table=test-table [id=8085f3ea29904b048db988bd461e0bcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:55:23.969079 14295 tablet_service.cc:1467] Processing CreateTablet for tablet 5f1e731a431940168737a5e6b789f29c (DEFAULT_TABLE table=test-table [id=8085f3ea29904b048db988bd461e0bcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:55:23.969357 14017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f1e731a431940168737a5e6b789f29c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:55:23.969587 14295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f1e731a431940168737a5e6b789f29c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:55:23.971477 14415 tablet_replica.cc:330] T c990c9b912044467a9d11499fbc303b2 P 0f15e3c842c043d6938d92d91a857968: stopping tablet replica
I20260925 04:55:23.971989 14415 raft_consensus.cc:2229] T c990c9b912044467a9d11499fbc303b2 P 0f15e3c842c043d6938d92d91a857968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:55:23.972486 14415 raft_consensus.cc:2258] T c990c9b912044467a9d11499fbc303b2 P 0f15e3c842c043d6938d92d91a857968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:55:23.973336 14414 tablet_replica.cc:330] T c990c9b912044467a9d11499fbc303b2 P 68fc8818ace94823866a0260adc36dcb: stopping tablet replica
I20260925 04:55:23.973763 14414 raft_consensus.cc:2229] T c990c9b912044467a9d11499fbc303b2 P 68fc8818ace94823866a0260adc36dcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:55:23.974274 14414 raft_consensus.cc:2258] T c990c9b912044467a9d11499fbc303b2 P 68fc8818ace94823866a0260adc36dcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:55:23.976115 14390 tablet_bootstrap.cc:492] T 5f1e731a431940168737a5e6b789f29c P 0f15e3c842c043d6938d92d91a857968: Bootstrap starting.
I20260925 04:55:23.976547 14157 tablet_service.cc:1467] Processing CreateTablet for tablet 5f1e731a431940168737a5e6b789f29c (DEFAULT_TABLE table=test-table [id=8085f3ea29904b048db988bd461e0bcc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:55:23.977021 14157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f1e731a431940168737a5e6b789f29c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:55:23.977722 14390 tablet_bootstrap.cc:654] T 5f1e731a431940168737a5e6b789f29c P 0f15e3c842c043d6938d92d91a857968: Neither blocks nor log segments found. Creating new log.
I20260925 04:55:23.979660 14390 tablet_bootstrap.cc:492] T 5f1e731a431940168737a5e6b789f29c P 0f15e3c842c043d6938d92d91a857968: No bootstrap required, opened a new log
I20260925 04:55:23.979815 14390 ts_tablet_manager.cc:1397] T 5f1e731a431940168737a5e6b789f29c P 0f15e3c842c043d6938d92d91a857968: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 04:55:23.980585 14390 raft_consensus.cc:348] T 5f1e731a431940168737a5e6b789f29c P 0f15e3c842c043d6938d92d91a857968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } }
I20260925 04:55:23.980823 14390 raft_consensus.cc:374] T 5f1e731a431940168737a5e6b789f29c P 0f15e3c842c043d6938d92d91a857968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:55:23.980921 14390 raft_consensus.cc:729] T 5f1e731a431940168737a5e6b789f29c P 0f15e3c842c043d6938d92d91a857968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f15e3c842c043d6938d92d91a857968, State: Initialized, Role: FOLLOWER
I20260925 04:55:23.981166 14390 consensus_queue.cc:260] T 5f1e731a431940168737a5e6b789f29c P 0f15e3c842c043d6938d92d91a857968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } }
I20260925 04:55:23.982046 14390 ts_tablet_manager.cc:1428] T 5f1e731a431940168737a5e6b789f29c P 0f15e3c842c043d6938d92d91a857968: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 04:55:23.982445 14415 ts_tablet_manager.cc:1905] T c990c9b912044467a9d11499fbc303b2 P 0f15e3c842c043d6938d92d91a857968: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:55:23.983950 14391 tablet_bootstrap.cc:492] T 5f1e731a431940168737a5e6b789f29c P 63173e501f9d48e5953533d6e2a73cff: Bootstrap starting.
I20260925 04:55:23.985327 14391 tablet_bootstrap.cc:654] T 5f1e731a431940168737a5e6b789f29c P 63173e501f9d48e5953533d6e2a73cff: Neither blocks nor log segments found. Creating new log.
I20260925 04:55:23.988873 14392 tablet_bootstrap.cc:492] T 5f1e731a431940168737a5e6b789f29c P 68fc8818ace94823866a0260adc36dcb: Bootstrap starting.
I20260925 04:55:23.989513 14415 ts_tablet_manager.cc:1918] T c990c9b912044467a9d11499fbc303b2 P 0f15e3c842c043d6938d92d91a857968: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:55:23.989648 14415 log.cc:1196] T c990c9b912044467a9d11499fbc303b2 P 0f15e3c842c043d6938d92d91a857968: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-0/wal/wals/c990c9b912044467a9d11499fbc303b2
I20260925 04:55:23.990178 14415 ts_tablet_manager.cc:1939] T c990c9b912044467a9d11499fbc303b2 P 0f15e3c842c043d6938d92d91a857968: Deleting consensus metadata
I20260925 04:55:23.990568 14392 tablet_bootstrap.cc:654] T 5f1e731a431940168737a5e6b789f29c P 68fc8818ace94823866a0260adc36dcb: Neither blocks nor log segments found. Creating new log.
I20260925 04:55:23.991987 13865 catalog_manager.cc:4841] TS 0f15e3c842c043d6938d92d91a857968 (127.6.29.129:39065): tablet c990c9b912044467a9d11499fbc303b2 (table test-table [id=ab57c3745d564208907fe9133d16a9d7]) successfully deleted
I20260925 04:55:23.992550 14413 ts_tablet_manager.cc:1905] T c990c9b912044467a9d11499fbc303b2 P 63173e501f9d48e5953533d6e2a73cff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:55:23.993904 14414 ts_tablet_manager.cc:1905] T c990c9b912044467a9d11499fbc303b2 P 68fc8818ace94823866a0260adc36dcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:55:23.995711 14391 tablet_bootstrap.cc:492] T 5f1e731a431940168737a5e6b789f29c P 63173e501f9d48e5953533d6e2a73cff: No bootstrap required, opened a new log
I20260925 04:55:23.995923 14391 ts_tablet_manager.cc:1397] T 5f1e731a431940168737a5e6b789f29c P 63173e501f9d48e5953533d6e2a73cff: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.001s
I20260925 04:55:23.997051 14391 raft_consensus.cc:348] T 5f1e731a431940168737a5e6b789f29c P 63173e501f9d48e5953533d6e2a73cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } }
I20260925 04:55:23.997355 14391 raft_consensus.cc:374] T 5f1e731a431940168737a5e6b789f29c P 63173e501f9d48e5953533d6e2a73cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:55:23.997474 14391 raft_consensus.cc:729] T 5f1e731a431940168737a5e6b789f29c P 63173e501f9d48e5953533d6e2a73cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63173e501f9d48e5953533d6e2a73cff, State: Initialized, Role: FOLLOWER
I20260925 04:55:23.997830 14391 consensus_queue.cc:260] T 5f1e731a431940168737a5e6b789f29c P 63173e501f9d48e5953533d6e2a73cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } }
I20260925 04:55:23.998574 14391 ts_tablet_manager.cc:1428] T 5f1e731a431940168737a5e6b789f29c P 63173e501f9d48e5953533d6e2a73cff: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 04:55:24.004201 14414 ts_tablet_manager.cc:1918] T c990c9b912044467a9d11499fbc303b2 P 68fc8818ace94823866a0260adc36dcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:55:24.004391 14414 log.cc:1196] T c990c9b912044467a9d11499fbc303b2 P 68fc8818ace94823866a0260adc36dcb: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-2/wal/wals/c990c9b912044467a9d11499fbc303b2
I20260925 04:55:24.004997 14414 ts_tablet_manager.cc:1939] T c990c9b912044467a9d11499fbc303b2 P 68fc8818ace94823866a0260adc36dcb: Deleting consensus metadata
I20260925 04:55:24.006877 13867 catalog_manager.cc:4841] TS 68fc8818ace94823866a0260adc36dcb (127.6.29.131:43831): tablet c990c9b912044467a9d11499fbc303b2 (table test-table [id=ab57c3745d564208907fe9133d16a9d7]) successfully deleted
I20260925 04:55:24.008462 14392 tablet_bootstrap.cc:492] T 5f1e731a431940168737a5e6b789f29c P 68fc8818ace94823866a0260adc36dcb: No bootstrap required, opened a new log
I20260925 04:55:24.008617 14392 ts_tablet_manager.cc:1397] T 5f1e731a431940168737a5e6b789f29c P 68fc8818ace94823866a0260adc36dcb: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.011s
I20260925 04:55:24.009443 14392 raft_consensus.cc:348] T 5f1e731a431940168737a5e6b789f29c P 68fc8818ace94823866a0260adc36dcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } }
I20260925 04:55:24.009668 14392 raft_consensus.cc:374] T 5f1e731a431940168737a5e6b789f29c P 68fc8818ace94823866a0260adc36dcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:55:24.009763 14392 raft_consensus.cc:729] T 5f1e731a431940168737a5e6b789f29c P 68fc8818ace94823866a0260adc36dcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68fc8818ace94823866a0260adc36dcb, State: Initialized, Role: FOLLOWER
I20260925 04:55:24.010015 14392 consensus_queue.cc:260] T 5f1e731a431940168737a5e6b789f29c P 68fc8818ace94823866a0260adc36dcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } }
I20260925 04:55:24.010586 14392 ts_tablet_manager.cc:1428] T 5f1e731a431940168737a5e6b789f29c P 68fc8818ace94823866a0260adc36dcb: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 04:55:24.012481 14413 ts_tablet_manager.cc:1918] T c990c9b912044467a9d11499fbc303b2 P 63173e501f9d48e5953533d6e2a73cff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:55:24.012704 14413 log.cc:1196] T c990c9b912044467a9d11499fbc303b2 P 63173e501f9d48e5953533d6e2a73cff: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-1/wal/wals/c990c9b912044467a9d11499fbc303b2
I20260925 04:55:24.013286 14413 ts_tablet_manager.cc:1939] T c990c9b912044467a9d11499fbc303b2 P 63173e501f9d48e5953533d6e2a73cff: Deleting consensus metadata
I20260925 04:55:24.015027 13866 catalog_manager.cc:4841] TS 63173e501f9d48e5953533d6e2a73cff (127.6.29.130:45827): tablet c990c9b912044467a9d11499fbc303b2 (table test-table [id=ab57c3745d564208907fe9133d16a9d7]) successfully deleted
I20260925 04:55:24.058555 14396 raft_consensus.cc:482] T 5f1e731a431940168737a5e6b789f29c P 68fc8818ace94823866a0260adc36dcb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:55:24.058861 14396 raft_consensus.cc:504] T 5f1e731a431940168737a5e6b789f29c P 68fc8818ace94823866a0260adc36dcb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } }
I20260925 04:55:24.060859 14396 leader_election.cc:288] T 5f1e731a431940168737a5e6b789f29c P 68fc8818ace94823866a0260adc36dcb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f15e3c842c043d6938d92d91a857968 (127.6.29.129:39065), 63173e501f9d48e5953533d6e2a73cff (127.6.29.130:45827)
I20260925 04:55:24.070274 14385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:24.061534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:53951 (local address 127.6.29.130:45827)
0925 04:55:24.061712 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:24.061717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:24.062382 (+ 665us) server_negotiation.cc:407] Connection header received
0925 04:55:24.062522 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:24.062528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:24.062604 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:24.062711 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:55:24.063269 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:24.064457 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:24.067149 (+ 2692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:24.068599 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:24.068764 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:24.068848 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:24.069783 (+ 935us) server_negotiation.cc:299] Negotiation successful
0925 04:55:24.069925 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 04:55:24.070286 14419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:24.061092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:45827 (local address 127.6.29.131:53951)
0925 04:55:24.062269 (+ 1177us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:24.062286 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:24.062409 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:24.062749 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:24.062754 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:24.062778 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:55:24.063093 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:24.063106 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:24.064600 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:24.064606 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:24.066969 (+ 2363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:24.066986 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:24.069393 (+ 2407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:24.069399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:24.069588 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:24.069623 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 04:55:24.069731 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 04:55:24.069877 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1033,"mutex_wait_us":575,"thread_start_us":270,"threads_started":1}
I20260925 04:55:24.071450 14255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:24.061092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:39065 (local address 127.6.29.131:42907)
0925 04:55:24.061392 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:24.061413 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:24.061541 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:24.062146 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:24.062150 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:24.062177 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:55:24.062404 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:24.062414 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:24.063779 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:24.063786 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:24.065531 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:24.065548 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:24.070691 (+ 5143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:24.070697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:24.070894 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:24.070929 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 04:55:24.071019 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 04:55:24.071178 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":1}
I20260925 04:55:24.071461 14177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f1e731a431940168737a5e6b789f29c" candidate_uuid: "68fc8818ace94823866a0260adc36dcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63173e501f9d48e5953533d6e2a73cff" is_pre_election: true
I20260925 04:55:24.071995 14177 raft_consensus.cc:2454] T 5f1e731a431940168737a5e6b789f29c P 63173e501f9d48e5953533d6e2a73cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68fc8818ace94823866a0260adc36dcb in term 0.
I20260925 04:55:24.072207 14389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:24.061330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:42907 (local address 127.6.29.129:39065)
0925 04:55:24.061506 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:24.061512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:24.061534 (+ 22us) server_negotiation.cc:407] Connection header received
0925 04:55:24.061729 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:24.061737 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:24.061858 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:24.061956 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:55:24.062576 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:24.063617 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:24.067598 (+ 3981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:24.068891 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:24.069063 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:24.069144 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:24.071683 (+ 2539us) server_negotiation.cc:299] Negotiation successful
0925 04:55:24.071913 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 04:55:24.072731 14248 leader_election.cc:302] T 5f1e731a431940168737a5e6b789f29c P 68fc8818ace94823866a0260adc36dcb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63173e501f9d48e5953533d6e2a73cff, 68fc8818ace94823866a0260adc36dcb; no voters:
I20260925 04:55:24.072863 14042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f1e731a431940168737a5e6b789f29c" candidate_uuid: "68fc8818ace94823866a0260adc36dcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f15e3c842c043d6938d92d91a857968" is_pre_election: true
I20260925 04:55:24.073207 14396 raft_consensus.cc:2781] T 5f1e731a431940168737a5e6b789f29c P 68fc8818ace94823866a0260adc36dcb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:55:24.073194 14042 raft_consensus.cc:2454] T 5f1e731a431940168737a5e6b789f29c P 0f15e3c842c043d6938d92d91a857968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68fc8818ace94823866a0260adc36dcb in term 0.
I20260925 04:55:24.073320 14396 raft_consensus.cc:482] T 5f1e731a431940168737a5e6b789f29c P 68fc8818ace94823866a0260adc36dcb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:55:24.073406 14396 raft_consensus.cc:3037] T 5f1e731a431940168737a5e6b789f29c P 68fc8818ace94823866a0260adc36dcb [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:55:24.076563 14396 raft_consensus.cc:504] T 5f1e731a431940168737a5e6b789f29c P 68fc8818ace94823866a0260adc36dcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } }
I20260925 04:55:24.077289 14396 leader_election.cc:288] T 5f1e731a431940168737a5e6b789f29c P 68fc8818ace94823866a0260adc36dcb [CANDIDATE]: Term 1 election: Requested vote from peers 0f15e3c842c043d6938d92d91a857968 (127.6.29.129:39065), 63173e501f9d48e5953533d6e2a73cff (127.6.29.130:45827)
I20260925 04:55:24.077886 14042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f1e731a431940168737a5e6b789f29c" candidate_uuid: "68fc8818ace94823866a0260adc36dcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f15e3c842c043d6938d92d91a857968"
I20260925 04:55:24.078106 14042 raft_consensus.cc:3037] T 5f1e731a431940168737a5e6b789f29c P 0f15e3c842c043d6938d92d91a857968 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:55:24.078320 14177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f1e731a431940168737a5e6b789f29c" candidate_uuid: "68fc8818ace94823866a0260adc36dcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63173e501f9d48e5953533d6e2a73cff"
I20260925 04:55:24.078533 14177 raft_consensus.cc:3037] T 5f1e731a431940168737a5e6b789f29c P 63173e501f9d48e5953533d6e2a73cff [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:55:24.081132 14042 raft_consensus.cc:2454] T 5f1e731a431940168737a5e6b789f29c P 0f15e3c842c043d6938d92d91a857968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68fc8818ace94823866a0260adc36dcb in term 1.
I20260925 04:55:24.082113 14177 raft_consensus.cc:2454] T 5f1e731a431940168737a5e6b789f29c P 63173e501f9d48e5953533d6e2a73cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68fc8818ace94823866a0260adc36dcb in term 1.
I20260925 04:55:24.082667 14248 leader_election.cc:302] T 5f1e731a431940168737a5e6b789f29c P 68fc8818ace94823866a0260adc36dcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63173e501f9d48e5953533d6e2a73cff, 68fc8818ace94823866a0260adc36dcb; no voters:
I20260925 04:55:24.083061 14396 raft_consensus.cc:2781] T 5f1e731a431940168737a5e6b789f29c P 68fc8818ace94823866a0260adc36dcb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:55:24.083485 14396 raft_consensus.cc:686] T 5f1e731a431940168737a5e6b789f29c P 68fc8818ace94823866a0260adc36dcb [term 1 LEADER]: Becoming Leader. State: Replica: 68fc8818ace94823866a0260adc36dcb, State: Running, Role: LEADER
I20260925 04:55:24.083936 14396 consensus_queue.cc:237] T 5f1e731a431940168737a5e6b789f29c P 68fc8818ace94823866a0260adc36dcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } }
I20260925 04:55:24.087641 13879 catalog_manager.cc:5495] T 5f1e731a431940168737a5e6b789f29c P 68fc8818ace94823866a0260adc36dcb reported cstate change: term changed from 0 to 1, leader changed from <none> to 68fc8818ace94823866a0260adc36dcb (127.6.29.131). New cstate: current_term: 1 leader_uuid: "68fc8818ace94823866a0260adc36dcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } health_report { overall_health: HEALTHY } } }
Sep 25 04:55:24 dist-test-slave-xzjr krb5kdc[13836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312121, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.29.190@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM
I20260925 04:55:24.159351 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:24.142521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37234 (local address 127.6.29.190:34931)
0925 04:55:24.142817 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:24.142827 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:24.143359 (+ 532us) server_negotiation.cc:407] Connection header received
0925 04:55:24.143546 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:24.143551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:24.143631 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:24.143757 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:24.145415 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:24.146180 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:24.148091 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:24.148495 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:24.148591 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:24.152903 (+ 4312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:24.152909 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:24.152913 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:24.152952 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:24.156020 (+ 3068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:24.156953 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:24.156958 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:24.156961 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:24.157119 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:24.157540 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:24.157544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:24.157548 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:24.157957 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:24.158143 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:24.158594 (+ 451us) server_negotiation.cc:299] Negotiation successful
0925 04:55:24.158964 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":177,"server-negotiator.queue_time_us":134}
I20260925 04:55:24.159373 14427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:24.142497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:34931 (local address 127.0.0.1:37234)
0925 04:55:24.143219 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:24.143240 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:24.143408 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:24.143857 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:24.143862 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:24.144855 (+ 993us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:24.145212 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:24.145224 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:24.146312 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:24.146318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:24.147946 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:24.147960 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:24.148683 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:24.148690 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:24.148858 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:24.149633 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:24.149656 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:24.152606 (+ 2950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:24.156139 (+ 3533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:24.156154 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:24.156162 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:24.156801 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:24.157231 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:24.157237 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:24.157240 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:24.157408 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:24.158102 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:24.158111 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:24.158317 (+ 206us) client_negotiation.cc:769] Sending connection context
0925 04:55:24.158499 (+ 182us) client_negotiation.cc:240] Negotiation successful
0925 04:55:24.158843 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":408,"threads_started":1}
I20260925 04:55:24.164649 13879 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:37234:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 04:55:24.165812 13879 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:55:24.173200 14017 tablet_service.cc:1467] Processing CreateTablet for tablet a5380a11142f40499afbd334ebf5680c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=61abe174f580471e8b3f7741b54c8464]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 04:55:24.173182 14157 tablet_service.cc:1467] Processing CreateTablet for tablet a5380a11142f40499afbd334ebf5680c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=61abe174f580471e8b3f7741b54c8464]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 04:55:24.173529 14295 tablet_service.cc:1467] Processing CreateTablet for tablet a5380a11142f40499afbd334ebf5680c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=61abe174f580471e8b3f7741b54c8464]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 04:55:24.173744 14017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5380a11142f40499afbd334ebf5680c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:55:24.173767 14157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5380a11142f40499afbd334ebf5680c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:55:24.174015 14295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5380a11142f40499afbd334ebf5680c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:55:24.181224 14391 tablet_bootstrap.cc:492] T a5380a11142f40499afbd334ebf5680c P 63173e501f9d48e5953533d6e2a73cff: Bootstrap starting.
I20260925 04:55:24.181221 14390 tablet_bootstrap.cc:492] T a5380a11142f40499afbd334ebf5680c P 0f15e3c842c043d6938d92d91a857968: Bootstrap starting.
I20260925 04:55:24.181268 14392 tablet_bootstrap.cc:492] T a5380a11142f40499afbd334ebf5680c P 68fc8818ace94823866a0260adc36dcb: Bootstrap starting.
I20260925 04:55:24.182916 14392 tablet_bootstrap.cc:654] T a5380a11142f40499afbd334ebf5680c P 68fc8818ace94823866a0260adc36dcb: Neither blocks nor log segments found. Creating new log.
I20260925 04:55:24.182989 14390 tablet_bootstrap.cc:654] T a5380a11142f40499afbd334ebf5680c P 0f15e3c842c043d6938d92d91a857968: Neither blocks nor log segments found. Creating new log.
I20260925 04:55:24.182991 14391 tablet_bootstrap.cc:654] T a5380a11142f40499afbd334ebf5680c P 63173e501f9d48e5953533d6e2a73cff: Neither blocks nor log segments found. Creating new log.
I20260925 04:55:24.193856 14391 tablet_bootstrap.cc:492] T a5380a11142f40499afbd334ebf5680c P 63173e501f9d48e5953533d6e2a73cff: No bootstrap required, opened a new log
I20260925 04:55:24.194077 14391 ts_tablet_manager.cc:1397] T a5380a11142f40499afbd334ebf5680c P 63173e501f9d48e5953533d6e2a73cff: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.001s
I20260925 04:55:24.194949 14391 raft_consensus.cc:348] T a5380a11142f40499afbd334ebf5680c P 63173e501f9d48e5953533d6e2a73cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } }
I20260925 04:55:24.195672 14390 tablet_bootstrap.cc:492] T a5380a11142f40499afbd334ebf5680c P 0f15e3c842c043d6938d92d91a857968: No bootstrap required, opened a new log
I20260925 04:55:24.195919 14390 ts_tablet_manager.cc:1397] T a5380a11142f40499afbd334ebf5680c P 0f15e3c842c043d6938d92d91a857968: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.005s
I20260925 04:55:24.196681 14390 raft_consensus.cc:348] T a5380a11142f40499afbd334ebf5680c P 0f15e3c842c043d6938d92d91a857968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } }
I20260925 04:55:24.196913 14390 raft_consensus.cc:374] T a5380a11142f40499afbd334ebf5680c P 0f15e3c842c043d6938d92d91a857968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:55:24.196990 14390 raft_consensus.cc:729] T a5380a11142f40499afbd334ebf5680c P 0f15e3c842c043d6938d92d91a857968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f15e3c842c043d6938d92d91a857968, State: Initialized, Role: FOLLOWER
I20260925 04:55:24.197283 14390 consensus_queue.cc:260] T a5380a11142f40499afbd334ebf5680c P 0f15e3c842c043d6938d92d91a857968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } }
I20260925 04:55:24.197564 14392 tablet_bootstrap.cc:492] T a5380a11142f40499afbd334ebf5680c P 68fc8818ace94823866a0260adc36dcb: No bootstrap required, opened a new log
I20260925 04:55:24.197741 14392 ts_tablet_manager.cc:1397] T a5380a11142f40499afbd334ebf5680c P 68fc8818ace94823866a0260adc36dcb: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.005s
I20260925 04:55:24.197731 14397 tablet_replica.cc:439] T a5380a11142f40499afbd334ebf5680c P 0f15e3c842c043d6938d92d91a857968: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } } }
I20260925 04:55:24.198100 14397 tablet_replica.cc:442] T a5380a11142f40499afbd334ebf5680c P 0f15e3c842c043d6938d92d91a857968: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:55:24.195217 14391 raft_consensus.cc:374] T a5380a11142f40499afbd334ebf5680c P 63173e501f9d48e5953533d6e2a73cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:55:24.198406 14391 raft_consensus.cc:729] T a5380a11142f40499afbd334ebf5680c P 63173e501f9d48e5953533d6e2a73cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63173e501f9d48e5953533d6e2a73cff, State: Initialized, Role: FOLLOWER
I20260925 04:55:24.199283 14392 raft_consensus.cc:348] T a5380a11142f40499afbd334ebf5680c P 68fc8818ace94823866a0260adc36dcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } }
I20260925 04:55:24.199545 14392 raft_consensus.cc:374] T a5380a11142f40499afbd334ebf5680c P 68fc8818ace94823866a0260adc36dcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:55:24.199623 14392 raft_consensus.cc:729] T a5380a11142f40499afbd334ebf5680c P 68fc8818ace94823866a0260adc36dcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68fc8818ace94823866a0260adc36dcb, State: Initialized, Role: FOLLOWER
I20260925 04:55:24.200011 14391 consensus_queue.cc:260] T a5380a11142f40499afbd334ebf5680c P 63173e501f9d48e5953533d6e2a73cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } }
I20260925 04:55:24.200331 14390 ts_tablet_manager.cc:1428] T a5380a11142f40499afbd334ebf5680c P 0f15e3c842c043d6938d92d91a857968: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260925 04:55:24.200477 14401 tablet_replica.cc:439] T a5380a11142f40499afbd334ebf5680c P 63173e501f9d48e5953533d6e2a73cff: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } } }
I20260925 04:55:24.200822 14401 tablet_replica.cc:442] T a5380a11142f40499afbd334ebf5680c P 63173e501f9d48e5953533d6e2a73cff: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:55:24.201841 14391 ts_tablet_manager.cc:1428] T a5380a11142f40499afbd334ebf5680c P 63173e501f9d48e5953533d6e2a73cff: Time spent starting tablet: real 0.008s user 0.000s sys 0.003s
I20260925 04:55:24.202183 14392 consensus_queue.cc:260] T a5380a11142f40499afbd334ebf5680c P 68fc8818ace94823866a0260adc36dcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } }
I20260925 04:55:24.202622 14396 tablet_replica.cc:439] T a5380a11142f40499afbd334ebf5680c P 68fc8818ace94823866a0260adc36dcb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } } }
I20260925 04:55:24.202899 14396 tablet_replica.cc:442] T a5380a11142f40499afbd334ebf5680c P 68fc8818ace94823866a0260adc36dcb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:55:24.203882 14392 ts_tablet_manager.cc:1428] T a5380a11142f40499afbd334ebf5680c P 68fc8818ace94823866a0260adc36dcb: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20260925 04:55:24.497165 14396 consensus_queue.cc:1035] T 5f1e731a431940168737a5e6b789f29c P 68fc8818ace94823866a0260adc36dcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:55:24.501530 14396 raft_consensus.cc:482] T a5380a11142f40499afbd334ebf5680c P 68fc8818ace94823866a0260adc36dcb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:55:24.501741 14396 raft_consensus.cc:504] T a5380a11142f40499afbd334ebf5680c P 68fc8818ace94823866a0260adc36dcb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } }
I20260925 04:55:24.502516 14396 leader_election.cc:288] T a5380a11142f40499afbd334ebf5680c P 68fc8818ace94823866a0260adc36dcb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f15e3c842c043d6938d92d91a857968 (127.6.29.129:39065), 63173e501f9d48e5953533d6e2a73cff (127.6.29.130:45827)
I20260925 04:55:24.503582 14042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5380a11142f40499afbd334ebf5680c" candidate_uuid: "68fc8818ace94823866a0260adc36dcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f15e3c842c043d6938d92d91a857968" is_pre_election: true
I20260925 04:55:24.503846 14042 raft_consensus.cc:2454] T a5380a11142f40499afbd334ebf5680c P 0f15e3c842c043d6938d92d91a857968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68fc8818ace94823866a0260adc36dcb in term 0.
I20260925 04:55:24.504546 14247 leader_election.cc:302] T a5380a11142f40499afbd334ebf5680c P 68fc8818ace94823866a0260adc36dcb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f15e3c842c043d6938d92d91a857968, 68fc8818ace94823866a0260adc36dcb; no voters:
I20260925 04:55:24.504469 14420 consensus_queue.cc:1035] T 5f1e731a431940168737a5e6b789f29c P 68fc8818ace94823866a0260adc36dcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:55:24.504516 14177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5380a11142f40499afbd334ebf5680c" candidate_uuid: "68fc8818ace94823866a0260adc36dcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63173e501f9d48e5953533d6e2a73cff" is_pre_election: true
I20260925 04:55:24.504901 14420 raft_consensus.cc:2781] T a5380a11142f40499afbd334ebf5680c P 68fc8818ace94823866a0260adc36dcb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:55:24.504892 14177 raft_consensus.cc:2454] T a5380a11142f40499afbd334ebf5680c P 63173e501f9d48e5953533d6e2a73cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68fc8818ace94823866a0260adc36dcb in term 0.
I20260925 04:55:24.505029 14420 raft_consensus.cc:482] T a5380a11142f40499afbd334ebf5680c P 68fc8818ace94823866a0260adc36dcb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:55:24.505124 14420 raft_consensus.cc:3037] T a5380a11142f40499afbd334ebf5680c P 68fc8818ace94823866a0260adc36dcb [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:55:24.509012 14420 raft_consensus.cc:504] T a5380a11142f40499afbd334ebf5680c P 68fc8818ace94823866a0260adc36dcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } }
I20260925 04:55:24.509832 14420 leader_election.cc:288] T a5380a11142f40499afbd334ebf5680c P 68fc8818ace94823866a0260adc36dcb [CANDIDATE]: Term 1 election: Requested vote from peers 0f15e3c842c043d6938d92d91a857968 (127.6.29.129:39065), 63173e501f9d48e5953533d6e2a73cff (127.6.29.130:45827)
I20260925 04:55:24.510133 14043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5380a11142f40499afbd334ebf5680c" candidate_uuid: "68fc8818ace94823866a0260adc36dcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f15e3c842c043d6938d92d91a857968"
I20260925 04:55:24.510133 14177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5380a11142f40499afbd334ebf5680c" candidate_uuid: "68fc8818ace94823866a0260adc36dcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63173e501f9d48e5953533d6e2a73cff"
I20260925 04:55:24.510344 14043 raft_consensus.cc:3037] T a5380a11142f40499afbd334ebf5680c P 0f15e3c842c043d6938d92d91a857968 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:55:24.510367 14177 raft_consensus.cc:3037] T a5380a11142f40499afbd334ebf5680c P 63173e501f9d48e5953533d6e2a73cff [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:55:24.513324 14177 raft_consensus.cc:2454] T a5380a11142f40499afbd334ebf5680c P 63173e501f9d48e5953533d6e2a73cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68fc8818ace94823866a0260adc36dcb in term 1.
I20260925 04:55:24.513346 14043 raft_consensus.cc:2454] T a5380a11142f40499afbd334ebf5680c P 0f15e3c842c043d6938d92d91a857968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68fc8818ace94823866a0260adc36dcb in term 1.
I20260925 04:55:24.513698 14247 leader_election.cc:302] T a5380a11142f40499afbd334ebf5680c P 68fc8818ace94823866a0260adc36dcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f15e3c842c043d6938d92d91a857968, 68fc8818ace94823866a0260adc36dcb; no voters:
I20260925 04:55:24.514010 14420 raft_consensus.cc:2781] T a5380a11142f40499afbd334ebf5680c P 68fc8818ace94823866a0260adc36dcb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:55:24.514185 14420 raft_consensus.cc:686] T a5380a11142f40499afbd334ebf5680c P 68fc8818ace94823866a0260adc36dcb [term 1 LEADER]: Becoming Leader. State: Replica: 68fc8818ace94823866a0260adc36dcb, State: Running, Role: LEADER
I20260925 04:55:24.514448 14420 consensus_queue.cc:237] T a5380a11142f40499afbd334ebf5680c P 68fc8818ace94823866a0260adc36dcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } }
I20260925 04:55:24.516017 14433 tablet_replica.cc:439] T a5380a11142f40499afbd334ebf5680c P 68fc8818ace94823866a0260adc36dcb: TxnStatusTablet state changed. Reason: New leader 68fc8818ace94823866a0260adc36dcb. Latest consensus state: current_term: 1 leader_uuid: "68fc8818ace94823866a0260adc36dcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } } }
I20260925 04:55:24.516309 14433 tablet_replica.cc:442] T a5380a11142f40499afbd334ebf5680c P 68fc8818ace94823866a0260adc36dcb: This TxnStatusTablet replica's current role is: LEADER
I20260925 04:55:24.516855 14439 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 04:55:24.517079 14439 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 04:55:24.517386 13879 catalog_manager.cc:5495] T a5380a11142f40499afbd334ebf5680c P 68fc8818ace94823866a0260adc36dcb reported cstate change: term changed from 0 to 1, leader changed from <none> to 68fc8818ace94823866a0260adc36dcb (127.6.29.131). New cstate: current_term: 1 leader_uuid: "68fc8818ace94823866a0260adc36dcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } health_report { overall_health: HEALTHY } } }
I20260925 04:55:24.572639 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:24.565623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37740 (local address 127.6.29.131:43831)
0925 04:55:24.566495 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:24.566499 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:24.566522 (+ 23us) server_negotiation.cc:407] Connection header received
0925 04:55:24.566600 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:24.566604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:24.566677 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:24.566876 (+ 199us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:55:24.567326 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:24.568236 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:24.570194 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:24.570628 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:24.570763 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:24.571283 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:55:24.572005 (+ 722us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:55:24.572116 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:24.572257 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 04:55:24.572323 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":753,"thread_start_us":571,"threads_started":1}
I20260925 04:55:24.572671 14427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:24.565499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.131:43831 (local address 127.0.0.1:37740)
0925 04:55:24.565749 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:24.565771 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:24.565920 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:24.566909 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:24.566914 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:24.566928 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:55:24.567162 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:24.567172 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:24.568361 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:24.568367 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:24.570054 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:24.570069 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:24.570802 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:24.570809 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:24.571017 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:24.571097 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:55:24.572109 (+ 1012us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:55:24.572131 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:55:24.572232 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 04:55:24.572397 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 04:55:24.576550 14177 raft_consensus.cc:1261] T a5380a11142f40499afbd334ebf5680c P 63173e501f9d48e5953533d6e2a73cff [term 1 FOLLOWER]: Refusing update from remote peer 68fc8818ace94823866a0260adc36dcb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:55:24.576550 14043 raft_consensus.cc:1261] T a5380a11142f40499afbd334ebf5680c P 0f15e3c842c043d6938d92d91a857968 [term 1 FOLLOWER]: Refusing update from remote peer 68fc8818ace94823866a0260adc36dcb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:55:24.577204 14433 consensus_queue.cc:1035] T a5380a11142f40499afbd334ebf5680c P 68fc8818ace94823866a0260adc36dcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:55:24.577531 14420 consensus_queue.cc:1035] T a5380a11142f40499afbd334ebf5680c P 68fc8818ace94823866a0260adc36dcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:55:24.588567 14397 tablet_replica.cc:439] T a5380a11142f40499afbd334ebf5680c P 0f15e3c842c043d6938d92d91a857968: TxnStatusTablet state changed. Reason: New leader 68fc8818ace94823866a0260adc36dcb. Latest consensus state: current_term: 1 leader_uuid: "68fc8818ace94823866a0260adc36dcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } } }
I20260925 04:55:24.588886 14397 tablet_replica.cc:442] T a5380a11142f40499afbd334ebf5680c P 0f15e3c842c043d6938d92d91a857968: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:55:24.590428 14432 mvcc.cc:205] Tried to move back new op lower bound from 7333118462259445760 to 7333118462013321216. Current Snapshot: MvccSnapshot[applied={T|T < 7333118462259445760}]
I20260925 04:55:24.592476 14397 tablet_replica.cc:439] T a5380a11142f40499afbd334ebf5680c P 0f15e3c842c043d6938d92d91a857968: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "68fc8818ace94823866a0260adc36dcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } } }
I20260925 04:55:24.592772 14397 tablet_replica.cc:442] T a5380a11142f40499afbd334ebf5680c P 0f15e3c842c043d6938d92d91a857968: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:55:24.594036 14401 tablet_replica.cc:439] T a5380a11142f40499afbd334ebf5680c P 63173e501f9d48e5953533d6e2a73cff: TxnStatusTablet state changed. Reason: New leader 68fc8818ace94823866a0260adc36dcb. Latest consensus state: current_term: 1 leader_uuid: "68fc8818ace94823866a0260adc36dcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } } }
I20260925 04:55:24.594321 14401 tablet_replica.cc:442] T a5380a11142f40499afbd334ebf5680c P 63173e501f9d48e5953533d6e2a73cff: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:55:24.596772 14435 tablet_replica.cc:439] T a5380a11142f40499afbd334ebf5680c P 68fc8818ace94823866a0260adc36dcb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "68fc8818ace94823866a0260adc36dcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } } }
I20260925 04:55:24.597383 14433 tablet_replica.cc:439] T a5380a11142f40499afbd334ebf5680c P 68fc8818ace94823866a0260adc36dcb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "68fc8818ace94823866a0260adc36dcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } } }
I20260925 04:55:24.597697 14433 tablet_replica.cc:442] T a5380a11142f40499afbd334ebf5680c P 68fc8818ace94823866a0260adc36dcb: This TxnStatusTablet replica's current role is: LEADER
I20260925 04:55:24.597903 14435 tablet_replica.cc:442] T a5380a11142f40499afbd334ebf5680c P 68fc8818ace94823866a0260adc36dcb: This TxnStatusTablet replica's current role is: LEADER
I20260925 04:55:24.617439 14401 tablet_replica.cc:439] T a5380a11142f40499afbd334ebf5680c P 63173e501f9d48e5953533d6e2a73cff: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "68fc8818ace94823866a0260adc36dcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } } }
I20260925 04:55:24.617748 14401 tablet_replica.cc:442] T a5380a11142f40499afbd334ebf5680c P 63173e501f9d48e5953533d6e2a73cff: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 04:55:24.626008 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:24.616385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37746 (local address 127.6.29.131:43831)
0925 04:55:24.616574 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:24.616581 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:24.619406 (+ 2825us) server_negotiation.cc:407] Connection header received
0925 04:55:24.619755 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:24.619784 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:24.619869 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:24.619931 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:55:24.620683 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:24.621884 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:24.623881 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:24.624574 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:24.624707 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:24.625197 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:55:24.625356 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:55:24.625445 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:24.625617 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 04:55:24.625700 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 04:55:24.642504 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:24.636452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:36189 (local address 127.6.29.131:43831)
0925 04:55:24.636661 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:24.636670 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:24.637263 (+ 593us) server_negotiation.cc:407] Connection header received
0925 04:55:24.637478 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:24.637483 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:24.637575 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:24.637673 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:55:24.638183 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:24.638969 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:24.640593 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:24.641053 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:24.641193 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:24.641622 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:55:24.641716 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:55:24.641828 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:24.641970 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 04:55:24.642086 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":57}
I20260925 04:55:24.642593 14452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:24.636458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.131:43831 (local address 127.6.29.131:36189)
0925 04:55:24.637169 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:24.637184 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:24.637345 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:24.637683 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:24.637688 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:24.637697 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:55:24.638032 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:24.638040 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:24.639114 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:24.639119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:24.640459 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:24.640473 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:24.641235 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:24.641243 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:24.641375 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:24.641430 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:55:24.641828 (+ 398us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:55:24.641840 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 04:55:24.641952 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 04:55:24.642131 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"mutex_wait_us":26,"thread_start_us":386,"threads_started":1}
I20260925 04:55:24.705407 13880 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:55:24.705685 13880 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37224:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:55:24.708635 13880 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 94caa156203345ea9f4f18a77e0f320f: Sending DeleteTablet for 3 replicas of tablet 5f1e731a431940168737a5e6b789f29c
I20260925 04:55:24.709380 14017 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f1e731a431940168737a5e6b789f29c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:55:24 UTC) from {username='oryx', principal='oryx/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:35994
I20260925 04:55:24.709617 14296 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f1e731a431940168737a5e6b789f29c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:55:24 UTC) from {username='oryx', principal='oryx/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:37732
I20260925 04:55:24.709797 14157 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f1e731a431940168737a5e6b789f29c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:55:24 UTC) from {username='oryx', principal='oryx/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:48120
I20260925 04:55:24.720024 14456 tablet_replica.cc:330] T 5f1e731a431940168737a5e6b789f29c P 63173e501f9d48e5953533d6e2a73cff: stopping tablet replica
I20260925 04:55:24.720338 14456 raft_consensus.cc:2229] T 5f1e731a431940168737a5e6b789f29c P 63173e501f9d48e5953533d6e2a73cff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:55:24.720621 14456 raft_consensus.cc:2258] T 5f1e731a431940168737a5e6b789f29c P 63173e501f9d48e5953533d6e2a73cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:55:24.723042 14456 ts_tablet_manager.cc:1905] T 5f1e731a431940168737a5e6b789f29c P 63173e501f9d48e5953533d6e2a73cff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:55:24.727262 14454 tablet_replica.cc:330] T 5f1e731a431940168737a5e6b789f29c P 0f15e3c842c043d6938d92d91a857968: stopping tablet replica
I20260925 04:55:24.727481 14454 raft_consensus.cc:2229] T 5f1e731a431940168737a5e6b789f29c P 0f15e3c842c043d6938d92d91a857968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:55:24.727732 14454 raft_consensus.cc:2258] T 5f1e731a431940168737a5e6b789f29c P 0f15e3c842c043d6938d92d91a857968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:55:24.728471 14455 tablet_replica.cc:330] T 5f1e731a431940168737a5e6b789f29c P 68fc8818ace94823866a0260adc36dcb: stopping tablet replica
I20260925 04:55:24.728696 14455 raft_consensus.cc:2229] T 5f1e731a431940168737a5e6b789f29c P 68fc8818ace94823866a0260adc36dcb [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:55:24.729002 14454 ts_tablet_manager.cc:1905] T 5f1e731a431940168737a5e6b789f29c P 0f15e3c842c043d6938d92d91a857968: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:55:24.729393 14455 raft_consensus.cc:2258] T 5f1e731a431940168737a5e6b789f29c P 68fc8818ace94823866a0260adc36dcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:55:24.730383 14455 ts_tablet_manager.cc:1905] T 5f1e731a431940168737a5e6b789f29c P 68fc8818ace94823866a0260adc36dcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:55:24.736086 14454 ts_tablet_manager.cc:1918] T 5f1e731a431940168737a5e6b789f29c P 0f15e3c842c043d6938d92d91a857968: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:55:24.736240 14454 log.cc:1196] T 5f1e731a431940168737a5e6b789f29c P 0f15e3c842c043d6938d92d91a857968: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-0/wal/wals/5f1e731a431940168737a5e6b789f29c
I20260925 04:55:24.736689 14454 ts_tablet_manager.cc:1939] T 5f1e731a431940168737a5e6b789f29c P 0f15e3c842c043d6938d92d91a857968: Deleting consensus metadata
I20260925 04:55:24.737748 14455 ts_tablet_manager.cc:1918] T 5f1e731a431940168737a5e6b789f29c P 68fc8818ace94823866a0260adc36dcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:55:24.737915 14455 log.cc:1196] T 5f1e731a431940168737a5e6b789f29c P 68fc8818ace94823866a0260adc36dcb: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-2/wal/wals/5f1e731a431940168737a5e6b789f29c
I20260925 04:55:24.737905 13865 catalog_manager.cc:4841] TS 0f15e3c842c043d6938d92d91a857968 (127.6.29.129:39065): tablet 5f1e731a431940168737a5e6b789f29c (table test-table [id=8085f3ea29904b048db988bd461e0bcc]) successfully deleted
I20260925 04:55:24.738390 14455 ts_tablet_manager.cc:1939] T 5f1e731a431940168737a5e6b789f29c P 68fc8818ace94823866a0260adc36dcb: Deleting consensus metadata
I20260925 04:55:24.739483 13867 catalog_manager.cc:4841] TS 68fc8818ace94823866a0260adc36dcb (127.6.29.131:43831): tablet 5f1e731a431940168737a5e6b789f29c (table test-table [id=8085f3ea29904b048db988bd461e0bcc]) successfully deleted
I20260925 04:55:24.742064 14456 ts_tablet_manager.cc:1918] T 5f1e731a431940168737a5e6b789f29c P 63173e501f9d48e5953533d6e2a73cff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:55:24.742275 14456 log.cc:1196] T 5f1e731a431940168737a5e6b789f29c P 63173e501f9d48e5953533d6e2a73cff: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-1/wal/wals/5f1e731a431940168737a5e6b789f29c
I20260925 04:55:24.742722 14456 ts_tablet_manager.cc:1939] T 5f1e731a431940168737a5e6b789f29c P 63173e501f9d48e5953533d6e2a73cff: Deleting consensus metadata
I20260925 04:55:24.743961 13866 catalog_manager.cc:4841] TS 63173e501f9d48e5953533d6e2a73cff (127.6.29.130:45827): tablet 5f1e731a431940168737a5e6b789f29c (table test-table [id=8085f3ea29904b048db988bd461e0bcc]) successfully deleted
Sep 25 04:55:24 dist-test-slave-xzjr krb5kdc[13836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 04:55:24 dist-test-slave-xzjr krb5kdc[13836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:24 dist-test-slave-xzjr krb5kdc[13836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
W20260925 04:55:24.782042 14463 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:24.770863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37236 (local address 127.6.29.190:34931)
0925 04:55:24.771364 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:24.771369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:24.771812 (+ 443us) server_negotiation.cc:407] Connection header received
0925 04:55:24.772058 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:24.772062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:24.772128 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:24.772245 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:24.773612 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:24.774356 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:24.775650 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:24.776148 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:24.776288 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:24.781328 (+ 5040us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37236: BlockingRecv error: recv got EOF from 127.0.0.1:37236 (error 108)
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":286,"threads_started":1}
Sep 25 04:55:24 dist-test-slave-xzjr krb5kdc[13836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM
I20260925 04:55:24.799889 14463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:24.784484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37250 (local address 127.6.29.190:34931)
0925 04:55:24.784665 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:24.784671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:24.784986 (+ 315us) server_negotiation.cc:407] Connection header received
0925 04:55:24.785145 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:24.785152 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:24.785245 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:24.785358 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:24.786532 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:24.787648 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:24.789047 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:24.789672 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:24.789803 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:24.794134 (+ 4331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:24.794139 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:24.794143 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:24.794180 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:24.796891 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:24.797737 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:24.797741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:24.797744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:24.797889 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:24.798335 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:24.798339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:24.798341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:24.798669 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:24.798993 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:24.799316 (+ 323us) server_negotiation.cc:299] Negotiation successful
0925 04:55:24.799558 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 04:55:24.803661 13880 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:55:24.804363 13880 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:55:24.811301 14017 tablet_service.cc:1467] Processing CreateTablet for tablet 8126991bbddb46a49ee05a6bb01b73b1 (DEFAULT_TABLE table=test-table [id=c87e8e21fe07496b87f1a4fe5592230e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:55:24.811826 14017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8126991bbddb46a49ee05a6bb01b73b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:55:24.811899 14296 tablet_service.cc:1467] Processing CreateTablet for tablet 8126991bbddb46a49ee05a6bb01b73b1 (DEFAULT_TABLE table=test-table [id=c87e8e21fe07496b87f1a4fe5592230e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:55:24.812317 14296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8126991bbddb46a49ee05a6bb01b73b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:55:24.816504 14157 tablet_service.cc:1467] Processing CreateTablet for tablet 8126991bbddb46a49ee05a6bb01b73b1 (DEFAULT_TABLE table=test-table [id=c87e8e21fe07496b87f1a4fe5592230e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:55:24.816941 14157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8126991bbddb46a49ee05a6bb01b73b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:55:24.823419 14470 tablet_bootstrap.cc:492] T 8126991bbddb46a49ee05a6bb01b73b1 P 0f15e3c842c043d6938d92d91a857968: Bootstrap starting.
I20260925 04:55:24.825611 14470 tablet_bootstrap.cc:654] T 8126991bbddb46a49ee05a6bb01b73b1 P 0f15e3c842c043d6938d92d91a857968: Neither blocks nor log segments found. Creating new log.
I20260925 04:55:24.827795 14471 tablet_bootstrap.cc:492] T 8126991bbddb46a49ee05a6bb01b73b1 P 68fc8818ace94823866a0260adc36dcb: Bootstrap starting.
I20260925 04:55:24.830083 14471 tablet_bootstrap.cc:654] T 8126991bbddb46a49ee05a6bb01b73b1 P 68fc8818ace94823866a0260adc36dcb: Neither blocks nor log segments found. Creating new log.
I20260925 04:55:24.834214 14470 tablet_bootstrap.cc:492] T 8126991bbddb46a49ee05a6bb01b73b1 P 0f15e3c842c043d6938d92d91a857968: No bootstrap required, opened a new log
I20260925 04:55:24.834448 14470 ts_tablet_manager.cc:1397] T 8126991bbddb46a49ee05a6bb01b73b1 P 0f15e3c842c043d6938d92d91a857968: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.008s
I20260925 04:55:24.835320 14470 raft_consensus.cc:348] T 8126991bbddb46a49ee05a6bb01b73b1 P 0f15e3c842c043d6938d92d91a857968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } }
I20260925 04:55:24.835748 14470 raft_consensus.cc:374] T 8126991bbddb46a49ee05a6bb01b73b1 P 0f15e3c842c043d6938d92d91a857968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:55:24.835922 14470 raft_consensus.cc:729] T 8126991bbddb46a49ee05a6bb01b73b1 P 0f15e3c842c043d6938d92d91a857968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f15e3c842c043d6938d92d91a857968, State: Initialized, Role: FOLLOWER
I20260925 04:55:24.836272 14470 consensus_queue.cc:260] T 8126991bbddb46a49ee05a6bb01b73b1 P 0f15e3c842c043d6938d92d91a857968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } }
I20260925 04:55:24.837582 14470 ts_tablet_manager.cc:1428] T 8126991bbddb46a49ee05a6bb01b73b1 P 0f15e3c842c043d6938d92d91a857968: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260925 04:55:24.841216 14471 tablet_bootstrap.cc:492] T 8126991bbddb46a49ee05a6bb01b73b1 P 68fc8818ace94823866a0260adc36dcb: No bootstrap required, opened a new log
I20260925 04:55:24.841388 14471 ts_tablet_manager.cc:1397] T 8126991bbddb46a49ee05a6bb01b73b1 P 68fc8818ace94823866a0260adc36dcb: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20260925 04:55:24.842538 14471 raft_consensus.cc:348] T 8126991bbddb46a49ee05a6bb01b73b1 P 68fc8818ace94823866a0260adc36dcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } }
I20260925 04:55:24.842772 14473 tablet_bootstrap.cc:492] T 8126991bbddb46a49ee05a6bb01b73b1 P 63173e501f9d48e5953533d6e2a73cff: Bootstrap starting.
I20260925 04:55:24.842837 14471 raft_consensus.cc:374] T 8126991bbddb46a49ee05a6bb01b73b1 P 68fc8818ace94823866a0260adc36dcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:55:24.842943 14471 raft_consensus.cc:729] T 8126991bbddb46a49ee05a6bb01b73b1 P 68fc8818ace94823866a0260adc36dcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68fc8818ace94823866a0260adc36dcb, State: Initialized, Role: FOLLOWER
I20260925 04:55:24.843202 14471 consensus_queue.cc:260] T 8126991bbddb46a49ee05a6bb01b73b1 P 68fc8818ace94823866a0260adc36dcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } }
I20260925 04:55:24.844023 14471 ts_tablet_manager.cc:1428] T 8126991bbddb46a49ee05a6bb01b73b1 P 68fc8818ace94823866a0260adc36dcb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 04:55:24.844861 14473 tablet_bootstrap.cc:654] T 8126991bbddb46a49ee05a6bb01b73b1 P 63173e501f9d48e5953533d6e2a73cff: Neither blocks nor log segments found. Creating new log.
I20260925 04:55:24.849313 14473 tablet_bootstrap.cc:492] T 8126991bbddb46a49ee05a6bb01b73b1 P 63173e501f9d48e5953533d6e2a73cff: No bootstrap required, opened a new log
I20260925 04:55:24.849491 14473 ts_tablet_manager.cc:1397] T 8126991bbddb46a49ee05a6bb01b73b1 P 63173e501f9d48e5953533d6e2a73cff: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260925 04:55:24.850224 14473 raft_consensus.cc:348] T 8126991bbddb46a49ee05a6bb01b73b1 P 63173e501f9d48e5953533d6e2a73cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } }
I20260925 04:55:24.850476 14473 raft_consensus.cc:374] T 8126991bbddb46a49ee05a6bb01b73b1 P 63173e501f9d48e5953533d6e2a73cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:55:24.850569 14473 raft_consensus.cc:729] T 8126991bbddb46a49ee05a6bb01b73b1 P 63173e501f9d48e5953533d6e2a73cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63173e501f9d48e5953533d6e2a73cff, State: Initialized, Role: FOLLOWER
I20260925 04:55:24.850842 14473 consensus_queue.cc:260] T 8126991bbddb46a49ee05a6bb01b73b1 P 63173e501f9d48e5953533d6e2a73cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } }
I20260925 04:55:24.851425 14473 ts_tablet_manager.cc:1428] T 8126991bbddb46a49ee05a6bb01b73b1 P 63173e501f9d48e5953533d6e2a73cff: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260925 04:55:25.092083 14420 raft_consensus.cc:482] T 8126991bbddb46a49ee05a6bb01b73b1 P 68fc8818ace94823866a0260adc36dcb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:55:25.092322 14420 raft_consensus.cc:504] T 8126991bbddb46a49ee05a6bb01b73b1 P 68fc8818ace94823866a0260adc36dcb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } }
I20260925 04:55:25.093093 14420 leader_election.cc:288] T 8126991bbddb46a49ee05a6bb01b73b1 P 68fc8818ace94823866a0260adc36dcb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f15e3c842c043d6938d92d91a857968 (127.6.29.129:39065), 63173e501f9d48e5953533d6e2a73cff (127.6.29.130:45827)
I20260925 04:55:25.093677 14177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8126991bbddb46a49ee05a6bb01b73b1" candidate_uuid: "68fc8818ace94823866a0260adc36dcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63173e501f9d48e5953533d6e2a73cff" is_pre_election: true
I20260925 04:55:25.093722 14043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8126991bbddb46a49ee05a6bb01b73b1" candidate_uuid: "68fc8818ace94823866a0260adc36dcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f15e3c842c043d6938d92d91a857968" is_pre_election: true
I20260925 04:55:25.093952 14177 raft_consensus.cc:2454] T 8126991bbddb46a49ee05a6bb01b73b1 P 63173e501f9d48e5953533d6e2a73cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68fc8818ace94823866a0260adc36dcb in term 0.
I20260925 04:55:25.093964 14043 raft_consensus.cc:2454] T 8126991bbddb46a49ee05a6bb01b73b1 P 0f15e3c842c043d6938d92d91a857968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68fc8818ace94823866a0260adc36dcb in term 0.
I20260925 04:55:25.094494 14248 leader_election.cc:302] T 8126991bbddb46a49ee05a6bb01b73b1 P 68fc8818ace94823866a0260adc36dcb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63173e501f9d48e5953533d6e2a73cff, 68fc8818ace94823866a0260adc36dcb; no voters:
I20260925 04:55:25.094797 14420 raft_consensus.cc:2781] T 8126991bbddb46a49ee05a6bb01b73b1 P 68fc8818ace94823866a0260adc36dcb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:55:25.094961 14420 raft_consensus.cc:482] T 8126991bbddb46a49ee05a6bb01b73b1 P 68fc8818ace94823866a0260adc36dcb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:55:25.095081 14420 raft_consensus.cc:3037] T 8126991bbddb46a49ee05a6bb01b73b1 P 68fc8818ace94823866a0260adc36dcb [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:55:25.097440 14420 raft_consensus.cc:504] T 8126991bbddb46a49ee05a6bb01b73b1 P 68fc8818ace94823866a0260adc36dcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } }
I20260925 04:55:25.098023 14420 leader_election.cc:288] T 8126991bbddb46a49ee05a6bb01b73b1 P 68fc8818ace94823866a0260adc36dcb [CANDIDATE]: Term 1 election: Requested vote from peers 0f15e3c842c043d6938d92d91a857968 (127.6.29.129:39065), 63173e501f9d48e5953533d6e2a73cff (127.6.29.130:45827)
I20260925 04:55:25.098609 14177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8126991bbddb46a49ee05a6bb01b73b1" candidate_uuid: "68fc8818ace94823866a0260adc36dcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63173e501f9d48e5953533d6e2a73cff"
I20260925 04:55:25.098652 14043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8126991bbddb46a49ee05a6bb01b73b1" candidate_uuid: "68fc8818ace94823866a0260adc36dcb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f15e3c842c043d6938d92d91a857968"
I20260925 04:55:25.098858 14043 raft_consensus.cc:3037] T 8126991bbddb46a49ee05a6bb01b73b1 P 0f15e3c842c043d6938d92d91a857968 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:55:25.098873 14177 raft_consensus.cc:3037] T 8126991bbddb46a49ee05a6bb01b73b1 P 63173e501f9d48e5953533d6e2a73cff [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:55:25.101306 14177 raft_consensus.cc:2454] T 8126991bbddb46a49ee05a6bb01b73b1 P 63173e501f9d48e5953533d6e2a73cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68fc8818ace94823866a0260adc36dcb in term 1.
I20260925 04:55:25.101747 14248 leader_election.cc:302] T 8126991bbddb46a49ee05a6bb01b73b1 P 68fc8818ace94823866a0260adc36dcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63173e501f9d48e5953533d6e2a73cff, 68fc8818ace94823866a0260adc36dcb; no voters:
I20260925 04:55:25.101915 14043 raft_consensus.cc:2454] T 8126991bbddb46a49ee05a6bb01b73b1 P 0f15e3c842c043d6938d92d91a857968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68fc8818ace94823866a0260adc36dcb in term 1.
I20260925 04:55:25.102097 14420 raft_consensus.cc:2781] T 8126991bbddb46a49ee05a6bb01b73b1 P 68fc8818ace94823866a0260adc36dcb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:55:25.102268 14420 raft_consensus.cc:686] T 8126991bbddb46a49ee05a6bb01b73b1 P 68fc8818ace94823866a0260adc36dcb [term 1 LEADER]: Becoming Leader. State: Replica: 68fc8818ace94823866a0260adc36dcb, State: Running, Role: LEADER
I20260925 04:55:25.102557 14420 consensus_queue.cc:237] T 8126991bbddb46a49ee05a6bb01b73b1 P 68fc8818ace94823866a0260adc36dcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } }
I20260925 04:55:25.105538 13880 catalog_manager.cc:5495] T 8126991bbddb46a49ee05a6bb01b73b1 P 68fc8818ace94823866a0260adc36dcb reported cstate change: term changed from 0 to 1, leader changed from <none> to 68fc8818ace94823866a0260adc36dcb (127.6.29.131). New cstate: current_term: 1 leader_uuid: "68fc8818ace94823866a0260adc36dcb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } health_report { overall_health: HEALTHY } } }
I20260925 04:55:25.124572 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:25.118779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37752 (local address 127.6.29.131:43831)
0925 04:55:25.118970 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:25.118974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:25.119186 (+ 212us) server_negotiation.cc:407] Connection header received
0925 04:55:25.119394 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:25.119397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:25.119462 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:25.119536 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:55:25.119977 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:25.120722 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:25.122666 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:25.123149 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:25.123251 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:25.123733 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:55:25.123921 (+ 188us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:55:25.123988 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:25.124181 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 04:55:25.124250 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 04:55:25.127417 14177 raft_consensus.cc:1261] T 8126991bbddb46a49ee05a6bb01b73b1 P 63173e501f9d48e5953533d6e2a73cff [term 1 FOLLOWER]: Refusing update from remote peer 68fc8818ace94823866a0260adc36dcb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:55:25.127460 14043 raft_consensus.cc:1261] T 8126991bbddb46a49ee05a6bb01b73b1 P 0f15e3c842c043d6938d92d91a857968 [term 1 FOLLOWER]: Refusing update from remote peer 68fc8818ace94823866a0260adc36dcb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:55:25.128090 14420 consensus_queue.cc:1035] T 8126991bbddb46a49ee05a6bb01b73b1 P 68fc8818ace94823866a0260adc36dcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:55:25.128419 14433 consensus_queue.cc:1035] T 8126991bbddb46a49ee05a6bb01b73b1 P 68fc8818ace94823866a0260adc36dcb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:55:25.133489 14434 mvcc.cc:205] Tried to move back new op lower bound from 7333118464515784704 to 7333118464422793216. Current Snapshot: MvccSnapshot[applied={T|T < 7333118464515784704}]
I20260925 04:55:25.142099 13881 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:55:25.142336 13881 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:55:25.145627 13881 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 94caa156203345ea9f4f18a77e0f320f: Sending DeleteTablet for 3 replicas of tablet 8126991bbddb46a49ee05a6bb01b73b1
I20260925 04:55:25.146564 14296 tablet_service.cc:1514] Processing DeleteTablet for tablet 8126991bbddb46a49ee05a6bb01b73b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:55:25 UTC) from {username='oryx', principal='oryx/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:37732
I20260925 04:55:25.146826 14017 tablet_service.cc:1514] Processing DeleteTablet for tablet 8126991bbddb46a49ee05a6bb01b73b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:55:25 UTC) from {username='oryx', principal='oryx/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:35994
I20260925 04:55:25.146905 14157 tablet_service.cc:1514] Processing DeleteTablet for tablet 8126991bbddb46a49ee05a6bb01b73b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:55:25 UTC) from {username='oryx', principal='oryx/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:48120
I20260925 04:55:25.147169 14456 tablet_replica.cc:330] T 8126991bbddb46a49ee05a6bb01b73b1 P 63173e501f9d48e5953533d6e2a73cff: stopping tablet replica
I20260925 04:55:25.147198 14455 tablet_replica.cc:330] T 8126991bbddb46a49ee05a6bb01b73b1 P 68fc8818ace94823866a0260adc36dcb: stopping tablet replica
I20260925 04:55:25.147372 14456 raft_consensus.cc:2229] T 8126991bbddb46a49ee05a6bb01b73b1 P 63173e501f9d48e5953533d6e2a73cff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:55:25.147390 14455 raft_consensus.cc:2229] T 8126991bbddb46a49ee05a6bb01b73b1 P 68fc8818ace94823866a0260adc36dcb [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:55:25.147593 14456 raft_consensus.cc:2258] T 8126991bbddb46a49ee05a6bb01b73b1 P 63173e501f9d48e5953533d6e2a73cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:55:25.147707 14455 raft_consensus.cc:2258] T 8126991bbddb46a49ee05a6bb01b73b1 P 68fc8818ace94823866a0260adc36dcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:55:25.148016 14454 tablet_replica.cc:330] T 8126991bbddb46a49ee05a6bb01b73b1 P 0f15e3c842c043d6938d92d91a857968: stopping tablet replica
I20260925 04:55:25.148216 14454 raft_consensus.cc:2229] T 8126991bbddb46a49ee05a6bb01b73b1 P 0f15e3c842c043d6938d92d91a857968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:55:25.148006 13880 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 04:55:25.148372 14454 raft_consensus.cc:2258] T 8126991bbddb46a49ee05a6bb01b73b1 P 0f15e3c842c043d6938d92d91a857968 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 04:55:25.148662 13880 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:55:25.149693 14455 ts_tablet_manager.cc:1905] T 8126991bbddb46a49ee05a6bb01b73b1 P 68fc8818ace94823866a0260adc36dcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:55:25.149680 14454 ts_tablet_manager.cc:1905] T 8126991bbddb46a49ee05a6bb01b73b1 P 0f15e3c842c043d6938d92d91a857968: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:55:25.156978 14454 ts_tablet_manager.cc:1918] T 8126991bbddb46a49ee05a6bb01b73b1 P 0f15e3c842c043d6938d92d91a857968: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:55:25.156903 14157 tablet_service.cc:1467] Processing CreateTablet for tablet f02906dc2fc04d7380811a8d560ded37 (DEFAULT_TABLE table=test-table [id=6d2b8d9996164026ac6222da0e3f146a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:55:25.157403 14157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f02906dc2fc04d7380811a8d560ded37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:55:25.157437 14455 ts_tablet_manager.cc:1918] T 8126991bbddb46a49ee05a6bb01b73b1 P 68fc8818ace94823866a0260adc36dcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:55:25.157373 14454 log.cc:1196] T 8126991bbddb46a49ee05a6bb01b73b1 P 0f15e3c842c043d6938d92d91a857968: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-0/wal/wals/8126991bbddb46a49ee05a6bb01b73b1
I20260925 04:55:25.157584 14455 log.cc:1196] T 8126991bbddb46a49ee05a6bb01b73b1 P 68fc8818ace94823866a0260adc36dcb: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-2/wal/wals/8126991bbddb46a49ee05a6bb01b73b1
I20260925 04:55:25.158072 14454 ts_tablet_manager.cc:1939] T 8126991bbddb46a49ee05a6bb01b73b1 P 0f15e3c842c043d6938d92d91a857968: Deleting consensus metadata
I20260925 04:55:25.158087 14455 ts_tablet_manager.cc:1939] T 8126991bbddb46a49ee05a6bb01b73b1 P 68fc8818ace94823866a0260adc36dcb: Deleting consensus metadata
I20260925 04:55:25.159742 14456 ts_tablet_manager.cc:1905] T 8126991bbddb46a49ee05a6bb01b73b1 P 63173e501f9d48e5953533d6e2a73cff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:55:25.160156 13865 catalog_manager.cc:4841] TS 0f15e3c842c043d6938d92d91a857968 (127.6.29.129:39065): tablet 8126991bbddb46a49ee05a6bb01b73b1 (table test-table [id=c87e8e21fe07496b87f1a4fe5592230e]) successfully deleted
I20260925 04:55:25.160696 14017 tablet_service.cc:1467] Processing CreateTablet for tablet f02906dc2fc04d7380811a8d560ded37 (DEFAULT_TABLE table=test-table [id=6d2b8d9996164026ac6222da0e3f146a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:55:25.161077 14017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f02906dc2fc04d7380811a8d560ded37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:55:25.164301 14473 tablet_bootstrap.cc:492] T f02906dc2fc04d7380811a8d560ded37 P 63173e501f9d48e5953533d6e2a73cff: Bootstrap starting.
I20260925 04:55:25.166080 14473 tablet_bootstrap.cc:654] T f02906dc2fc04d7380811a8d560ded37 P 63173e501f9d48e5953533d6e2a73cff: Neither blocks nor log segments found. Creating new log.
I20260925 04:55:25.167104 14456 ts_tablet_manager.cc:1918] T 8126991bbddb46a49ee05a6bb01b73b1 P 63173e501f9d48e5953533d6e2a73cff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:55:25.167363 14456 log.cc:1196] T 8126991bbddb46a49ee05a6bb01b73b1 P 63173e501f9d48e5953533d6e2a73cff: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-1/wal/wals/8126991bbddb46a49ee05a6bb01b73b1
I20260925 04:55:25.167372 14470 tablet_bootstrap.cc:492] T f02906dc2fc04d7380811a8d560ded37 P 0f15e3c842c043d6938d92d91a857968: Bootstrap starting.
I20260925 04:55:25.167701 13867 catalog_manager.cc:4841] TS 68fc8818ace94823866a0260adc36dcb (127.6.29.131:43831): tablet 8126991bbddb46a49ee05a6bb01b73b1 (table test-table [id=c87e8e21fe07496b87f1a4fe5592230e]) successfully deleted
I20260925 04:55:25.168004 14456 ts_tablet_manager.cc:1939] T 8126991bbddb46a49ee05a6bb01b73b1 P 63173e501f9d48e5953533d6e2a73cff: Deleting consensus metadata
I20260925 04:55:25.168576 14296 tablet_service.cc:1467] Processing CreateTablet for tablet f02906dc2fc04d7380811a8d560ded37 (DEFAULT_TABLE table=test-table [id=6d2b8d9996164026ac6222da0e3f146a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:55:25.168996 14470 tablet_bootstrap.cc:654] T f02906dc2fc04d7380811a8d560ded37 P 0f15e3c842c043d6938d92d91a857968: Neither blocks nor log segments found. Creating new log.
I20260925 04:55:25.169044 14296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f02906dc2fc04d7380811a8d560ded37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:55:25.173589 13866 catalog_manager.cc:4841] TS 63173e501f9d48e5953533d6e2a73cff (127.6.29.130:45827): tablet 8126991bbddb46a49ee05a6bb01b73b1 (table test-table [id=c87e8e21fe07496b87f1a4fe5592230e]) successfully deleted
I20260925 04:55:25.174789 14471 tablet_bootstrap.cc:492] T f02906dc2fc04d7380811a8d560ded37 P 68fc8818ace94823866a0260adc36dcb: Bootstrap starting.
I20260925 04:55:25.176409 14470 tablet_bootstrap.cc:492] T f02906dc2fc04d7380811a8d560ded37 P 0f15e3c842c043d6938d92d91a857968: No bootstrap required, opened a new log
I20260925 04:55:25.176486 14471 tablet_bootstrap.cc:654] T f02906dc2fc04d7380811a8d560ded37 P 68fc8818ace94823866a0260adc36dcb: Neither blocks nor log segments found. Creating new log.
I20260925 04:55:25.176610 14470 ts_tablet_manager.cc:1397] T f02906dc2fc04d7380811a8d560ded37 P 0f15e3c842c043d6938d92d91a857968: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20260925 04:55:25.177675 14470 raft_consensus.cc:348] T f02906dc2fc04d7380811a8d560ded37 P 0f15e3c842c043d6938d92d91a857968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } }
I20260925 04:55:25.178048 14470 raft_consensus.cc:374] T f02906dc2fc04d7380811a8d560ded37 P 0f15e3c842c043d6938d92d91a857968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:55:25.179076 14473 tablet_bootstrap.cc:492] T f02906dc2fc04d7380811a8d560ded37 P 63173e501f9d48e5953533d6e2a73cff: No bootstrap required, opened a new log
I20260925 04:55:25.179258 14473 ts_tablet_manager.cc:1397] T f02906dc2fc04d7380811a8d560ded37 P 63173e501f9d48e5953533d6e2a73cff: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.005s
I20260925 04:55:25.179234 14470 raft_consensus.cc:729] T f02906dc2fc04d7380811a8d560ded37 P 0f15e3c842c043d6938d92d91a857968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f15e3c842c043d6938d92d91a857968, State: Initialized, Role: FOLLOWER
I20260925 04:55:25.179569 14470 consensus_queue.cc:260] T f02906dc2fc04d7380811a8d560ded37 P 0f15e3c842c043d6938d92d91a857968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } }
I20260925 04:55:25.180073 14473 raft_consensus.cc:348] T f02906dc2fc04d7380811a8d560ded37 P 63173e501f9d48e5953533d6e2a73cff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } }
I20260925 04:55:25.180306 14473 raft_consensus.cc:374] T f02906dc2fc04d7380811a8d560ded37 P 63173e501f9d48e5953533d6e2a73cff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:55:25.180392 14473 raft_consensus.cc:729] T f02906dc2fc04d7380811a8d560ded37 P 63173e501f9d48e5953533d6e2a73cff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63173e501f9d48e5953533d6e2a73cff, State: Initialized, Role: FOLLOWER
I20260925 04:55:25.180678 14473 consensus_queue.cc:260] T f02906dc2fc04d7380811a8d560ded37 P 63173e501f9d48e5953533d6e2a73cff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } }
I20260925 04:55:25.183399 14473 ts_tablet_manager.cc:1428] T f02906dc2fc04d7380811a8d560ded37 P 63173e501f9d48e5953533d6e2a73cff: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260925 04:55:25.183700 14470 ts_tablet_manager.cc:1428] T f02906dc2fc04d7380811a8d560ded37 P 0f15e3c842c043d6938d92d91a857968: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260925 04:55:25.185948 14471 tablet_bootstrap.cc:492] T f02906dc2fc04d7380811a8d560ded37 P 68fc8818ace94823866a0260adc36dcb: No bootstrap required, opened a new log
I20260925 04:55:25.186147 14471 ts_tablet_manager.cc:1397] T f02906dc2fc04d7380811a8d560ded37 P 68fc8818ace94823866a0260adc36dcb: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260925 04:55:25.186975 14471 raft_consensus.cc:348] T f02906dc2fc04d7380811a8d560ded37 P 68fc8818ace94823866a0260adc36dcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } }
I20260925 04:55:25.187268 14471 raft_consensus.cc:374] T f02906dc2fc04d7380811a8d560ded37 P 68fc8818ace94823866a0260adc36dcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:55:25.187362 14471 raft_consensus.cc:729] T f02906dc2fc04d7380811a8d560ded37 P 68fc8818ace94823866a0260adc36dcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68fc8818ace94823866a0260adc36dcb, State: Initialized, Role: FOLLOWER
I20260925 04:55:25.187599 14471 consensus_queue.cc:260] T f02906dc2fc04d7380811a8d560ded37 P 68fc8818ace94823866a0260adc36dcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } }
I20260925 04:55:25.188148 14471 ts_tablet_manager.cc:1428] T f02906dc2fc04d7380811a8d560ded37 P 68fc8818ace94823866a0260adc36dcb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 04:55:25.354208 14397 raft_consensus.cc:482] T f02906dc2fc04d7380811a8d560ded37 P 0f15e3c842c043d6938d92d91a857968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:55:25.354481 14397 raft_consensus.cc:504] T f02906dc2fc04d7380811a8d560ded37 P 0f15e3c842c043d6938d92d91a857968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } }
I20260925 04:55:25.356338 14397 leader_election.cc:288] T f02906dc2fc04d7380811a8d560ded37 P 0f15e3c842c043d6938d92d91a857968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63173e501f9d48e5953533d6e2a73cff (127.6.29.130:45827), 68fc8818ace94823866a0260adc36dcb (127.6.29.131:43831)
I20260925 04:55:25.374451 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:25.357176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:58905 (local address 127.6.29.131:43831)
0925 04:55:25.357381 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:25.357387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:25.368298 (+ 10911us) server_negotiation.cc:407] Connection header received
0925 04:55:25.368472 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:25.368477 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:25.368578 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:25.368666 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:55:25.369327 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:25.370482 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:25.372495 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:25.373487 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:25.373650 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:25.373730 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:25.374011 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 04:55:25.374155 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 04:55:25.374475 14483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:25.356671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.131:43831 (local address 127.6.29.129:58905)
0925 04:55:25.368161 (+ 11490us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:25.368184 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:25.368333 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:25.368710 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:25.368716 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:25.368757 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:55:25.369137 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:25.369149 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:25.370654 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:25.370659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:25.372338 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:25.372350 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:25.373672 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:25.373678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:25.373837 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:25.373869 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 04:55:25.373960 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 04:55:25.374092 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11314,"mutex_wait_us":32,"thread_start_us":11123,"threads_started":1}
I20260925 04:55:25.375068 14321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f02906dc2fc04d7380811a8d560ded37" candidate_uuid: "0f15e3c842c043d6938d92d91a857968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68fc8818ace94823866a0260adc36dcb" is_pre_election: true
I20260925 04:55:25.375331 14321 raft_consensus.cc:2454] T f02906dc2fc04d7380811a8d560ded37 P 68fc8818ace94823866a0260adc36dcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f15e3c842c043d6938d92d91a857968 in term 0.
I20260925 04:55:25.375851 13970 leader_election.cc:302] T f02906dc2fc04d7380811a8d560ded37 P 0f15e3c842c043d6938d92d91a857968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f15e3c842c043d6938d92d91a857968, 68fc8818ace94823866a0260adc36dcb; no voters:
I20260925 04:55:25.376241 14484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:25.356596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:45827 (local address 127.6.29.129:33603)
0925 04:55:25.367009 (+ 10413us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:25.367044 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:25.367238 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:25.368993 (+ 1755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:25.368999 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:25.369033 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:55:25.369512 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:25.369525 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:25.370983 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:25.370989 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:25.373124 (+ 2135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:25.373138 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:25.375175 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:25.375179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:25.375349 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:25.375386 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 04:55:25.375508 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 04:55:25.375641 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10197,"mutex_wait_us":23,"thread_start_us":11818,"threads_started":1}
I20260925 04:55:25.376267 14397 raft_consensus.cc:2781] T f02906dc2fc04d7380811a8d560ded37 P 0f15e3c842c043d6938d92d91a857968 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:55:25.376464 14485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:25.356757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:33603 (local address 127.6.29.130:45827)
0925 04:55:25.368662 (+ 11905us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:25.368670 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:25.368686 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:55:25.368749 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:25.368755 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:25.368827 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:25.368939 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:55:25.369772 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:25.370834 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:25.373302 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:25.374457 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:25.375029 (+ 572us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:25.375080 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:25.375929 (+ 849us) server_negotiation.cc:299] Negotiation successful
0925 04:55:25.376078 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11768,"thread_start_us":12320,"threads_started":1}
I20260925 04:55:25.376547 14397 raft_consensus.cc:482] T f02906dc2fc04d7380811a8d560ded37 P 0f15e3c842c043d6938d92d91a857968 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:55:25.376633 14397 raft_consensus.cc:3037] T f02906dc2fc04d7380811a8d560ded37 P 0f15e3c842c043d6938d92d91a857968 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:55:25.376991 14177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f02906dc2fc04d7380811a8d560ded37" candidate_uuid: "0f15e3c842c043d6938d92d91a857968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63173e501f9d48e5953533d6e2a73cff" is_pre_election: true
I20260925 04:55:25.377270 14177 raft_consensus.cc:2454] T f02906dc2fc04d7380811a8d560ded37 P 63173e501f9d48e5953533d6e2a73cff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f15e3c842c043d6938d92d91a857968 in term 0.
I20260925 04:55:25.379658 14397 raft_consensus.cc:504] T f02906dc2fc04d7380811a8d560ded37 P 0f15e3c842c043d6938d92d91a857968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } }
I20260925 04:55:25.380373 14397 leader_election.cc:288] T f02906dc2fc04d7380811a8d560ded37 P 0f15e3c842c043d6938d92d91a857968 [CANDIDATE]: Term 1 election: Requested vote from peers 63173e501f9d48e5953533d6e2a73cff (127.6.29.130:45827), 68fc8818ace94823866a0260adc36dcb (127.6.29.131:43831)
I20260925 04:55:25.380858 14177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f02906dc2fc04d7380811a8d560ded37" candidate_uuid: "0f15e3c842c043d6938d92d91a857968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63173e501f9d48e5953533d6e2a73cff"
I20260925 04:55:25.381001 14321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f02906dc2fc04d7380811a8d560ded37" candidate_uuid: "0f15e3c842c043d6938d92d91a857968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68fc8818ace94823866a0260adc36dcb"
I20260925 04:55:25.381114 14177 raft_consensus.cc:3037] T f02906dc2fc04d7380811a8d560ded37 P 63173e501f9d48e5953533d6e2a73cff [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:55:25.381268 14321 raft_consensus.cc:3037] T f02906dc2fc04d7380811a8d560ded37 P 68fc8818ace94823866a0260adc36dcb [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:55:25.383689 14321 raft_consensus.cc:2454] T f02906dc2fc04d7380811a8d560ded37 P 68fc8818ace94823866a0260adc36dcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f15e3c842c043d6938d92d91a857968 in term 1.
I20260925 04:55:25.384017 14177 raft_consensus.cc:2454] T f02906dc2fc04d7380811a8d560ded37 P 63173e501f9d48e5953533d6e2a73cff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f15e3c842c043d6938d92d91a857968 in term 1.
I20260925 04:55:25.384097 13970 leader_election.cc:302] T f02906dc2fc04d7380811a8d560ded37 P 0f15e3c842c043d6938d92d91a857968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f15e3c842c043d6938d92d91a857968, 68fc8818ace94823866a0260adc36dcb; no voters:
I20260925 04:55:25.384375 14397 raft_consensus.cc:2781] T f02906dc2fc04d7380811a8d560ded37 P 0f15e3c842c043d6938d92d91a857968 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:55:25.384814 14397 raft_consensus.cc:686] T f02906dc2fc04d7380811a8d560ded37 P 0f15e3c842c043d6938d92d91a857968 [term 1 LEADER]: Becoming Leader. State: Replica: 0f15e3c842c043d6938d92d91a857968, State: Running, Role: LEADER
I20260925 04:55:25.385206 14397 consensus_queue.cc:237] T f02906dc2fc04d7380811a8d560ded37 P 0f15e3c842c043d6938d92d91a857968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } }
I20260925 04:55:25.389051 13880 catalog_manager.cc:5495] T f02906dc2fc04d7380811a8d560ded37 P 0f15e3c842c043d6938d92d91a857968 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f15e3c842c043d6938d92d91a857968 (127.6.29.129). New cstate: current_term: 1 leader_uuid: "0f15e3c842c043d6938d92d91a857968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f15e3c842c043d6938d92d91a857968" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 39065 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 } health_report { overall_health: UNKNOWN } } }
I20260925 04:55:25.432454 14488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:25.426682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35996 (local address 127.6.29.129:39065)
0925 04:55:25.427205 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:25.427210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:25.427229 (+ 19us) server_negotiation.cc:407] Connection header received
0925 04:55:25.427307 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:25.427310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:25.427379 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:25.427564 (+ 185us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:55:25.428036 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:25.428823 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:25.430163 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:25.430584 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:25.430723 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:25.431168 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:55:25.431752 (+ 584us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:55:25.431977 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:25.432037 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 04:55:25.432117 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":270,"threads_started":1}
I20260925 04:55:25.447929 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:25.442720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:50901 (local address 127.6.29.131:43831)
0925 04:55:25.442951 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:25.442958 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:25.443200 (+ 242us) server_negotiation.cc:407] Connection header received
0925 04:55:25.443428 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:25.443432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:25.443494 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:25.443585 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:55:25.444143 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:25.444826 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:25.446184 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:25.446615 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:25.446756 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:25.447183 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:55:25.447280 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:55:25.447371 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:25.447521 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 04:55:25.447596 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 04:55:25.448038 14491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:25.442331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.131:43831 (local address 127.6.29.129:50901)
0925 04:55:25.443099 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:25.443114 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:25.443289 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:25.443613 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:25.443617 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:25.443638 (+ 21us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:55:25.443988 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:25.443997 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:25.444963 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:25.444969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:25.446038 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:25.446049 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:25.446794 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:25.446801 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:25.446937 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:25.446993 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:55:25.447388 (+ 395us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:55:25.447406 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 04:55:25.447489 (+ 83us) client_negotiation.cc:240] Negotiation successful
0925 04:55:25.447621 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":374,"threads_started":1}
I20260925 04:55:25.456341 14177 raft_consensus.cc:1261] T f02906dc2fc04d7380811a8d560ded37 P 63173e501f9d48e5953533d6e2a73cff [term 1 FOLLOWER]: Refusing update from remote peer 0f15e3c842c043d6938d92d91a857968: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:55:25.456410 14321 raft_consensus.cc:1261] T f02906dc2fc04d7380811a8d560ded37 P 68fc8818ace94823866a0260adc36dcb [term 1 FOLLOWER]: Refusing update from remote peer 0f15e3c842c043d6938d92d91a857968: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:55:25.457338 14486 consensus_queue.cc:1035] T f02906dc2fc04d7380811a8d560ded37 P 0f15e3c842c043d6938d92d91a857968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68fc8818ace94823866a0260adc36dcb" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:55:25.457993 14397 consensus_queue.cc:1035] T f02906dc2fc04d7380811a8d560ded37 P 0f15e3c842c043d6938d92d91a857968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63173e501f9d48e5953533d6e2a73cff" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 45827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:55:25.489647 14488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:25.478144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:58351 (local address 127.6.29.129:39065)
0925 04:55:25.478310 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:25.478314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:25.484499 (+ 6185us) server_negotiation.cc:407] Connection header received
0925 04:55:25.484707 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:25.484712 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:25.484800 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:25.484894 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:55:25.485372 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:25.486104 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:25.487619 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:25.488271 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:25.488405 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:25.488873 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:55:25.489003 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:55:25.489095 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:25.489288 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 04:55:25.489372 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 04:55:25.489689 14496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:25.477967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:39065 (local address 127.6.29.131:58351)
0925 04:55:25.484375 (+ 6408us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:25.484393 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:25.484543 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:25.484946 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:25.484951 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:25.484964 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 04:55:25.485246 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:25.485254 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:25.486235 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:25.486239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:25.487419 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:25.487436 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:25.488471 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:25.488477 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:25.488660 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:25.488707 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 04:55:25.489144 (+ 437us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 04:55:25.489159 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 04:55:25.489246 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 04:55:25.489400 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6216,"thread_start_us":5981,"threads_started":1}
I20260925 04:55:25.522920 13880 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:55:25.523244 13880 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:55:25.526317 13880 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 94caa156203345ea9f4f18a77e0f320f: Sending DeleteTablet for 3 replicas of tablet f02906dc2fc04d7380811a8d560ded37
I20260925 04:55:25.527251 14157 tablet_service.cc:1514] Processing DeleteTablet for tablet f02906dc2fc04d7380811a8d560ded37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:55:25 UTC) from {username='oryx', principal='oryx/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:48120
I20260925 04:55:25.527268 14017 tablet_service.cc:1514] Processing DeleteTablet for tablet f02906dc2fc04d7380811a8d560ded37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:55:25 UTC) from {username='oryx', principal='oryx/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:35994
I20260925 04:55:25.527593 14456 tablet_replica.cc:330] T f02906dc2fc04d7380811a8d560ded37 P 63173e501f9d48e5953533d6e2a73cff: stopping tablet replica
I20260925 04:55:25.527606 14454 tablet_replica.cc:330] T f02906dc2fc04d7380811a8d560ded37 P 0f15e3c842c043d6938d92d91a857968: stopping tablet replica
I20260925 04:55:25.527812 14454 raft_consensus.cc:2229] T f02906dc2fc04d7380811a8d560ded37 P 0f15e3c842c043d6938d92d91a857968 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:55:25.527849 14456 raft_consensus.cc:2229] T f02906dc2fc04d7380811a8d560ded37 P 63173e501f9d48e5953533d6e2a73cff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:55:25.528055 14456 raft_consensus.cc:2258] T f02906dc2fc04d7380811a8d560ded37 P 63173e501f9d48e5953533d6e2a73cff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:55:25.528381 14454 raft_consensus.cc:2258] T f02906dc2fc04d7380811a8d560ded37 P 0f15e3c842c043d6938d92d91a857968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:55:25.528928 14296 tablet_service.cc:1514] Processing DeleteTablet for tablet f02906dc2fc04d7380811a8d560ded37 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:55:25 UTC) from {username='oryx', principal='oryx/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:37732
I20260925 04:55:25.529258 14455 tablet_replica.cc:330] T f02906dc2fc04d7380811a8d560ded37 P 68fc8818ace94823866a0260adc36dcb: stopping tablet replica
I20260925 04:55:25.529404 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 13952
I20260925 04:55:25.529435 14455 raft_consensus.cc:2229] T f02906dc2fc04d7380811a8d560ded37 P 68fc8818ace94823866a0260adc36dcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:55:25.529659 14455 raft_consensus.cc:2258] T f02906dc2fc04d7380811a8d560ded37 P 68fc8818ace94823866a0260adc36dcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:55:25.529846 14456 ts_tablet_manager.cc:1905] T f02906dc2fc04d7380811a8d560ded37 P 63173e501f9d48e5953533d6e2a73cff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:55:25.531517 14455 ts_tablet_manager.cc:1905] T f02906dc2fc04d7380811a8d560ded37 P 68fc8818ace94823866a0260adc36dcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:55:25.536618 14456 ts_tablet_manager.cc:1918] T f02906dc2fc04d7380811a8d560ded37 P 63173e501f9d48e5953533d6e2a73cff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:55:25.536798 14456 log.cc:1196] T f02906dc2fc04d7380811a8d560ded37 P 63173e501f9d48e5953533d6e2a73cff: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-1/wal/wals/f02906dc2fc04d7380811a8d560ded37
I20260925 04:55:25.537104 14454 ts_tablet_manager.cc:1905] T f02906dc2fc04d7380811a8d560ded37 P 0f15e3c842c043d6938d92d91a857968: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:55:25.537206 14456 ts_tablet_manager.cc:1939] T f02906dc2fc04d7380811a8d560ded37 P 63173e501f9d48e5953533d6e2a73cff: Deleting consensus metadata
I20260925 04:55:25.538271 13866 catalog_manager.cc:4841] TS 63173e501f9d48e5953533d6e2a73cff (127.6.29.130:45827): tablet f02906dc2fc04d7380811a8d560ded37 (table test-table [id=6d2b8d9996164026ac6222da0e3f146a]) successfully deleted
I20260925 04:55:25.544610 14455 ts_tablet_manager.cc:1918] T f02906dc2fc04d7380811a8d560ded37 P 68fc8818ace94823866a0260adc36dcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:55:25.544862 14455 log.cc:1196] T f02906dc2fc04d7380811a8d560ded37 P 68fc8818ace94823866a0260adc36dcb: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-2/wal/wals/f02906dc2fc04d7380811a8d560ded37
I20260925 04:55:25.545339 14455 ts_tablet_manager.cc:1939] T f02906dc2fc04d7380811a8d560ded37 P 68fc8818ace94823866a0260adc36dcb: Deleting consensus metadata
I20260925 04:55:25.546438 13867 catalog_manager.cc:4841] TS 68fc8818ace94823866a0260adc36dcb (127.6.29.131:43831): tablet f02906dc2fc04d7380811a8d560ded37 (table test-table [id=6d2b8d9996164026ac6222da0e3f146a]) successfully deleted
I20260925 04:55:25.547104 14454 ts_tablet_manager.cc:1918] T f02906dc2fc04d7380811a8d560ded37 P 0f15e3c842c043d6938d92d91a857968: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 04:55:25.547260 14454 log.cc:1196] T f02906dc2fc04d7380811a8d560ded37 P 0f15e3c842c043d6938d92d91a857968: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790312053666503-6262-0/minicluster-data/ts-0/wal/wals/f02906dc2fc04d7380811a8d560ded37
I20260925 04:55:25.547646 14454 ts_tablet_manager.cc:1939] T f02906dc2fc04d7380811a8d560ded37 P 0f15e3c842c043d6938d92d91a857968: Deleting consensus metadata
I20260925 04:55:25.548658 13865 catalog_manager.cc:4841] TS 0f15e3c842c043d6938d92d91a857968 (127.6.29.129:39065): tablet f02906dc2fc04d7380811a8d560ded37 (table test-table [id=6d2b8d9996164026ac6222da0e3f146a]) successfully deleted
Sep 25 04:55:25 dist-test-slave-xzjr krb5kdc[13836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.29.129@KRBTEST.COM
I20260925 04:55:25.566463 14488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:25.540108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36010 (local address 127.6.29.129:39065)
0925 04:55:25.540674 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:25.540682 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:25.546702 (+ 6020us) server_negotiation.cc:407] Connection header received
0925 04:55:25.546943 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:25.546949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:25.547047 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:25.547126 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:25.548917 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:25.550085 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:25.552004 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:25.552717 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:25.552815 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:25.558351 (+ 5536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:25.558368 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:25.558390 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:25.558431 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:25.562831 (+ 4400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:25.563690 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:25.563700 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:25.563704 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:25.563906 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:25.564359 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:25.564364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:25.564368 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:25.564883 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:25.565219 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:25.565803 (+ 584us) server_negotiation.cc:299] Negotiation successful
0925 04:55:25.566045 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414}
I20260925 04:55:25.568042 14083 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:55:25.682778 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 13952
I20260925 04:55:25.703351 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 14092
Sep 25 04:55:25 dist-test-slave-xzjr krb5kdc[13836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.29.130@KRBTEST.COM
I20260925 04:55:25.718616 14485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:25.704216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48146 (local address 127.6.29.130:45827)
0925 04:55:25.704416 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:25.704420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:25.704437 (+ 17us) server_negotiation.cc:407] Connection header received
0925 04:55:25.704493 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:25.704497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:25.704570 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:25.704670 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:25.705827 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:25.706532 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:25.707758 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:25.708213 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:25.708310 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:25.712382 (+ 4072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:25.712398 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:25.712416 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:25.712453 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:25.715542 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:25.716450 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:25.716458 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:25.716460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:25.716602 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:25.717035 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:25.717040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:25.717043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:25.717417 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:25.717764 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:25.718050 (+ 286us) server_negotiation.cc:299] Negotiation successful
0925 04:55:25.718284 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 04:55:25.719280 14222 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:55:25.840430 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 14092
I20260925 04:55:25.862999 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 14231
Sep 25 04:55:25 dist-test-slave-xzjr krb5kdc[13836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.29.131@KRBTEST.COM
I20260925 04:55:25.878046 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:25.863786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37766 (local address 127.6.29.131:43831)
0925 04:55:25.863946 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:25.863951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:25.863969 (+ 18us) server_negotiation.cc:407] Connection header received
0925 04:55:25.864180 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:25.864184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:25.864250 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:25.864345 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:25.865546 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:25.866333 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:25.867565 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:25.867996 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:25.868090 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:25.872413 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:25.872427 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:25.872445 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:25.872492 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:25.875393 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:25.876104 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:25.876111 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:25.876114 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:25.876246 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:25.876603 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:25.876606 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:25.876608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:25.876983 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:25.877213 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:25.877494 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 04:55:25.877681 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 04:55:25.878763 14361 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:55:26.022985 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 14231
I20260925 04:55:26.048302 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 13849
I20260925 04:55:26.049582 13921 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:55:26.166240 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 13849
2026-09-25T04:55:26Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6033 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:55:26 dist-test-slave-xzjr krb5kdc[14507](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:55:26 dist-test-slave-xzjr krb5kdc[14507](info): set up 2 sockets
Sep 25 04:55:26 dist-test-slave-xzjr krb5kdc[14507](info): commencing operation
krb5kdc: starting...
W20260925 04:55:27.281167 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.008s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 04:55:27 dist-test-slave-xzjr krb5kdc[14507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:55:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:55:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.29.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
I20260925 04:55:27.512061 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:40807
--webserver_interface=127.6.29.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:41159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:40807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:55:27.840499 14520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:55:27.840799 14520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:55:27.840974 14520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:55:27.848203 14520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:55:27.848321 14520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:55:27.848430 14520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:55:27.848543 14520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:55:27.848631 14520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:55:27.858793 14520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:41159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:40807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:40807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:55:27.860183 14520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:55:27.861567 14520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:55:27.873518 14525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:27.873701 14526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:27.873725 14528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:55:27.874727 14520 server_base.cc:1034] running on GCE node
I20260925 04:55:27.875456 14520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:55:27.876982 14520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:55:27.878187 14520 hybrid_clock.cc:648] HybridClock initialized: now 1790312127878155 us; error 33 us; skew 500 ppm
I20260925 04:55:27.878501 14520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T04:55:27Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1781 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:55:28 dist-test-slave-xzjr krb5kdc[14534](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:55:28 dist-test-slave-xzjr krb5kdc[14534](info): set up 2 sockets
Sep 25 04:55:28 dist-test-slave-xzjr krb5kdc[14534](info): commencing operation
krb5kdc: starting...
W20260925 04:55:29.077282 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.003s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 04:55:29 dist-test-slave-xzjr krb5kdc[14534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:55:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:55:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.29.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
I20260925 04:55:29.303864 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:43689
--webserver_interface=127.6.29.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:34783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:43689
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:55:29.637312 14547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:55:29.637626 14547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:55:29.637718 14547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:55:29.644910 14547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:55:29.645031 14547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:55:29.645078 14547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:55:29.645128 14547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:55:29.645172 14547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:55:29.655539 14547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:34783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:43689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:43689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:55:29.656952 14547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:55:29.658262 14547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:55:29.670322 14552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:29.670413 14555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:29.670380 14553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:55:29.671332 14547 server_base.cc:1034] running on GCE node
I20260925 04:55:29.672261 14547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:55:29.673830 14547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:55:29.675047 14547 hybrid_clock.cc:648] HybridClock initialized: now 1790312129674996 us; error 34 us; skew 500 ppm
I20260925 04:55:29.675468 14547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:55:29 dist-test-slave-xzjr krb5kdc[14534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312129, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:55:29.678668 14547 init.cc:377] Logged in from keytab as kudu/127.6.29.190@KRBTEST.COM (short username kudu)
I20260925 04:55:29.680269 14547 webserver.cc:460] Webserver started at http://127.6.29.190:42231/ using document root <none> and password file <none>
I20260925 04:55:29.680977 14547 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:29.681116 14547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:29.681428 14547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:55:29.684473 14547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "950bf69bcda84cdeb862a86d57ff0e6e"
format_stamp: "Formatted at 2026-09-25 04:55:29 on dist-test-slave-xzjr"
server_key: "70deee2cece127e576e264cefbf50c44"
server_key_iv: "d74ebef282b814efd8c98cb7f7681cbd"
server_key_version: "encryptionkey@0"
I20260925 04:55:29.685266 14547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "950bf69bcda84cdeb862a86d57ff0e6e"
format_stamp: "Formatted at 2026-09-25 04:55:29 on dist-test-slave-xzjr"
server_key: "70deee2cece127e576e264cefbf50c44"
server_key_iv: "d74ebef282b814efd8c98cb7f7681cbd"
server_key_version: "encryptionkey@0"
I20260925 04:55:29.690629 14547 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 04:55:29.694394 14562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:29.695992 14547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:55:29.696195 14547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "950bf69bcda84cdeb862a86d57ff0e6e"
format_stamp: "Formatted at 2026-09-25 04:55:29 on dist-test-slave-xzjr"
server_key: "70deee2cece127e576e264cefbf50c44"
server_key_iv: "d74ebef282b814efd8c98cb7f7681cbd"
server_key_version: "encryptionkey@0"
I20260925 04:55:29.696414 14547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:29.713476 14547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:55:29.717993 14547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:55:29.718284 14547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:55:29.736940 14547 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:43689
I20260925 04:55:29.736960 14613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:43689 every 8 connection(s)
I20260925 04:55:29.738300 14547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:55:29.741976 14614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:55:29.745414 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 14547
I20260925 04:55:29.745653 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 04:55:29.746443 6262 external_mini_cluster.cc:1428] Setting key 5af4c406c6cb0dcf5cc84ee4d1df266e
I20260925 04:55:29.754561 14614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e: Bootstrap starting.
I20260925 04:55:29.757671 14614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e: Neither blocks nor log segments found. Creating new log.
I20260925 04:55:29.758689 14614 log.cc:824] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e: Log is configured to *not* fsync() on all Append() calls
Sep 25 04:55:29 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:55:29.761838 14614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e: No bootstrap required, opened a new log
I20260925 04:55:29.769821 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:29.748705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49308 (local address 127.6.29.190:43689)
0925 04:55:29.749494 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:29.749506 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:29.749548 (+ 42us) server_negotiation.cc:407] Connection header received
0925 04:55:29.751163 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:29.751187 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:29.751613 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:29.752087 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:29.753508 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:29.754912 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:29.756257 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:29.756929 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:29.757127 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:29.761190 (+ 4063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:29.761205 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:29.761226 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:29.761268 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:29.765324 (+ 4056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:29.766016 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:29.766026 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:29.766035 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:29.766236 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:29.766571 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:29.766576 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:29.766580 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:29.767264 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:29.767598 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:29.767928 (+ 330us) server_negotiation.cc:299] Negotiation successful
0925 04:55:29.768370 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":342,"threads_started":1}
I20260925 04:55:29.772289 14614 raft_consensus.cc:348] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950bf69bcda84cdeb862a86d57ff0e6e" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43689 } }
I20260925 04:55:29.772660 14614 raft_consensus.cc:374] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:55:29.772789 14614 raft_consensus.cc:729] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 950bf69bcda84cdeb862a86d57ff0e6e, State: Initialized, Role: FOLLOWER
I20260925 04:55:29.773325 14614 consensus_queue.cc:260] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950bf69bcda84cdeb862a86d57ff0e6e" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43689 } }
I20260925 04:55:29.773563 14614 raft_consensus.cc:388] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:55:29.773706 14614 raft_consensus.cc:482] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:55:29.773883 14614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:55:29.776052 14614 raft_consensus.cc:504] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950bf69bcda84cdeb862a86d57ff0e6e" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43689 } }
I20260925 04:55:29.776535 14614 leader_election.cc:302] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 950bf69bcda84cdeb862a86d57ff0e6e; no voters:
I20260925 04:55:29.777124 14614 leader_election.cc:288] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:55:29.777251 14619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:55:29.777928 14619 raft_consensus.cc:686] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e [term 1 LEADER]: Becoming Leader. State: Replica: 950bf69bcda84cdeb862a86d57ff0e6e, State: Running, Role: LEADER
I20260925 04:55:29.778556 14619 consensus_queue.cc:237] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950bf69bcda84cdeb862a86d57ff0e6e" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43689 } }
I20260925 04:55:29.779345 14614 sys_catalog.cc:564] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:55:29.781849 14620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "950bf69bcda84cdeb862a86d57ff0e6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950bf69bcda84cdeb862a86d57ff0e6e" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43689 } } }
I20260925 04:55:29.782528 14620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e [sys.catalog]: This master's current role is: LEADER
I20260925 04:55:29.782254 14621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 950bf69bcda84cdeb862a86d57ff0e6e. Latest consensus state: current_term: 1 leader_uuid: "950bf69bcda84cdeb862a86d57ff0e6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950bf69bcda84cdeb862a86d57ff0e6e" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43689 } } }
I20260925 04:55:29.783324 14621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e [sys.catalog]: This master's current role is: LEADER
I20260925 04:55:29.786829 14625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:55:29.791401 14625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:55:29.800412 14625 catalog_manager.cc:1347] Generated new cluster ID: fdb075eb35a54d6ba1a6abc94e2cb8a2
I20260925 04:55:29.800550 14625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:55:29.815076 14625 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:55:29.816421 14625 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:55:29.826941 14625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e: Generated new TSK 0
I20260925 04:55:29.828030 14625 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.29.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab.
Entry for principal kudu/127.6.29.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab.
I20260925 04:55:29.888005 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.129:0
--local_ip_for_outbound_sockets=127.6.29.129
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:43689
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:34783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:55:30.231969 14640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:55:30.232291 14640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:55:30.232390 14640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:55:30.239806 14640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:55:30.239949 14640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:55:30.240072 14640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.129
I20260925 04:55:30.249904 14640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:34783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:43689
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.129
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:55:30.251168 14640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:55:30.252528 14640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:55:30.266389 14648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:30.266410 14645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:30.266295 14646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:55:30.266933 14640 server_base.cc:1034] running on GCE node
I20260925 04:55:30.268172 14640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:55:30.269351 14640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:55:30.270643 14640 hybrid_clock.cc:648] HybridClock initialized: now 1790312130270541 us; error 75 us; skew 500 ppm
I20260925 04:55:30.271032 14640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:55:30 dist-test-slave-xzjr krb5kdc[14534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312130, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:55:30.274477 14640 init.cc:377] Logged in from keytab as kudu/127.6.29.129@KRBTEST.COM (short username kudu)
I20260925 04:55:30.276039 14640 webserver.cc:460] Webserver started at http://127.6.29.129:41977/ using document root <none> and password file <none>
I20260925 04:55:30.276741 14640 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:30.276842 14640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:30.277092 14640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:55:30.280506 14640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-0/data/instance:
uuid: "57326b8230174889be5a88abde39fdd8"
format_stamp: "Formatted at 2026-09-25 04:55:30 on dist-test-slave-xzjr"
server_key: "c5db73f2dfba2406d879f1b0c56b8448"
server_key_iv: "9960e994b6d33a7b13d00cd9cabccfb3"
server_key_version: "encryptionkey@0"
I20260925 04:55:30.281311 14640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance:
uuid: "57326b8230174889be5a88abde39fdd8"
format_stamp: "Formatted at 2026-09-25 04:55:30 on dist-test-slave-xzjr"
server_key: "c5db73f2dfba2406d879f1b0c56b8448"
server_key_iv: "9960e994b6d33a7b13d00cd9cabccfb3"
server_key_version: "encryptionkey@0"
I20260925 04:55:30.286773 14640 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 04:55:30.290665 14655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:30.292320 14640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:55:30.292495 14640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-0/wal
uuid: "57326b8230174889be5a88abde39fdd8"
format_stamp: "Formatted at 2026-09-25 04:55:30 on dist-test-slave-xzjr"
server_key: "c5db73f2dfba2406d879f1b0c56b8448"
server_key_iv: "9960e994b6d33a7b13d00cd9cabccfb3"
server_key_version: "encryptionkey@0"
I20260925 04:55:30.292725 14640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:30.306953 14640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:55:30.311671 14640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:55:30.312067 14640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:55:30.313114 14640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:55:30.314527 14640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:55:30.314600 14640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:30.314719 14640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:55:30.314796 14640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:30.349223 14640 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.129:39113
I20260925 04:55:30.349258 14767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.129:39113 every 8 connection(s)
I20260925 04:55:30.350611 14640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
I20260925 04:55:30.350991 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 14640
I20260925 04:55:30.351212 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance
I20260925 04:55:30.352101 6262 external_mini_cluster.cc:1428] Setting key eff159d8f5900e2cf253db9aef41ae62
Sep 25 04:55:30 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312130, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:55:30.374971 14770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:30.353428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:55955 (local address 127.6.29.190:43689)
0925 04:55:30.353921 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:30.353927 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:30.355230 (+ 1303us) server_negotiation.cc:407] Connection header received
0925 04:55:30.358226 (+ 2996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:30.358231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:30.358309 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:30.358406 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:30.360720 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:30.361446 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:30.363377 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:30.363884 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:30.364030 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:30.367946 (+ 3916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:30.367952 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:30.367955 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:30.367988 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:30.371263 (+ 3275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:30.372237 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:30.372246 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:30.372252 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:30.372419 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:30.372878 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:30.372887 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:30.372894 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:30.373306 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:30.373489 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:30.374298 (+ 809us) server_negotiation.cc:299] Negotiation successful
0925 04:55:30.374557 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":268,"threads_started":1}
I20260925 04:55:30.375824 14771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:30.354191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.129:55955)
0925 04:55:30.355046 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:30.355097 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:30.357889 (+ 2792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:30.358721 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:30.358737 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:30.359559 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:30.360476 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:30.360505 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:30.361616 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:30.361624 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:30.363216 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:30.363229 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:30.364133 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:30.364142 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:30.364399 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:30.365131 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:30.365155 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:30.367650 (+ 2495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:30.371442 (+ 3792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:30.371452 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:30.371472 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:30.372053 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:30.372549 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:30.372553 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:30.372555 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:30.372746 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:30.373462 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:30.373471 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:30.373896 (+ 425us) client_negotiation.cc:769] Sending connection context
0925 04:55:30.374136 (+ 240us) client_negotiation.cc:240] Negotiation successful
0925 04:55:30.374554 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":310,"threads_started":1}
WARNING: no policy specified for kudu/127.6.29.130@KRBTEST.COM; defaulting to no policy
I20260925 04:55:30.377512 14768 heartbeater.cc:347] Connected to a master server at 127.6.29.190:43689
I20260925 04:55:30.377980 14768 heartbeater.cc:464] Registering TS with master...
I20260925 04:55:30.378751 14768 heartbeater.cc:511] Master 127.6.29.190:43689 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.130@KRBTEST.COM" created.
I20260925 04:55:30.381299 14579 ts_manager.cc:194] Registered new tserver with Master: 57326b8230174889be5a88abde39fdd8 (127.6.29.129:39113)
I20260925 04:55:30.382756 14579 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.129@KRBTEST.COM'} at 127.6.29.129:55955
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab.
Entry for principal kudu/127.6.29.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab.
I20260925 04:55:30.408293 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.130:0
--local_ip_for_outbound_sockets=127.6.29.130
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:43689
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:34783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:55:30.734680 14774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:55:30.734972 14774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:55:30.735139 14774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:55:30.746297 14774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:55:30.746456 14774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:55:30.746627 14774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.130
I20260925 04:55:30.758199 14774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:34783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:43689
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.130
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:55:30.759443 14774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:55:30.760756 14774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:55:30.773976 14782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:30.774009 14779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:30.773979 14780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:55:30.774706 14774 server_base.cc:1034] running on GCE node
I20260925 04:55:30.775305 14774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:55:30.776265 14774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:55:30.777495 14774 hybrid_clock.cc:648] HybridClock initialized: now 1790312130777454 us; error 48 us; skew 500 ppm
I20260925 04:55:30.777853 14774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:55:30 dist-test-slave-xzjr krb5kdc[14534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312130, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:55:30.781390 14774 init.cc:377] Logged in from keytab as kudu/127.6.29.130@KRBTEST.COM (short username kudu)
I20260925 04:55:30.782867 14774 webserver.cc:460] Webserver started at http://127.6.29.130:46287/ using document root <none> and password file <none>
I20260925 04:55:30.783514 14774 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:30.783610 14774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:30.783941 14774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:55:30.786744 14774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-1/data/instance:
uuid: "4708ca219dc54da7a937934abc8f1e2b"
format_stamp: "Formatted at 2026-09-25 04:55:30 on dist-test-slave-xzjr"
server_key: "d52dc349b5eece226531bdfb540d770d"
server_key_iv: "ae8b7e323df176ae9a79732b821806d6"
server_key_version: "encryptionkey@0"
I20260925 04:55:30.787459 14774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance:
uuid: "4708ca219dc54da7a937934abc8f1e2b"
format_stamp: "Formatted at 2026-09-25 04:55:30 on dist-test-slave-xzjr"
server_key: "d52dc349b5eece226531bdfb540d770d"
server_key_iv: "ae8b7e323df176ae9a79732b821806d6"
server_key_version: "encryptionkey@0"
I20260925 04:55:30.792819 14774 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260925 04:55:30.796237 14789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:30.797715 14774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 04:55:30.797878 14774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-1/wal
uuid: "4708ca219dc54da7a937934abc8f1e2b"
format_stamp: "Formatted at 2026-09-25 04:55:30 on dist-test-slave-xzjr"
server_key: "d52dc349b5eece226531bdfb540d770d"
server_key_iv: "ae8b7e323df176ae9a79732b821806d6"
server_key_version: "encryptionkey@0"
I20260925 04:55:30.798111 14774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:30.826087 14774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:55:30.830619 14774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:55:30.830929 14774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:55:30.831944 14774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:55:30.833238 14774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:55:30.833312 14774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:30.833459 14774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:55:30.833518 14774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:30.866307 14774 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.130:42343
I20260925 04:55:30.866354 14901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.130:42343 every 8 connection(s)
I20260925 04:55:30.867633 14774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
I20260925 04:55:30.870954 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 14774
I20260925 04:55:30.871160 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance
I20260925 04:55:30.871587 6262 external_mini_cluster.cc:1428] Setting key ff07e9639fc4e4084f1b97d17e275d27
Sep 25 04:55:30 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312130, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:55:30.889469 14770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:30.870272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:36685 (local address 127.6.29.190:43689)
0925 04:55:30.870459 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:30.870463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:30.871734 (+ 1271us) server_negotiation.cc:407] Connection header received
0925 04:55:30.873827 (+ 2093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:30.873833 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:30.873913 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:30.874028 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:30.875519 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:30.876290 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:30.877799 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:30.878235 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:30.878367 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:30.882337 (+ 3970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:30.882346 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:30.882354 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:30.882399 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:30.885982 (+ 3583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:30.886766 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:30.886775 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:30.886781 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:30.886941 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:30.887353 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:30.887360 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:30.887366 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:30.887757 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:30.887922 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:30.888671 (+ 749us) server_negotiation.cc:299] Negotiation successful
0925 04:55:30.888937 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 04:55:30.890419 14904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:30.870780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.130:36685)
0925 04:55:30.871583 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:30.871630 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:30.873574 (+ 1944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:30.874261 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:30.874269 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:30.874775 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:30.875359 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:30.875374 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:30.876438 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:30.876447 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:30.877658 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:30.877670 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:30.878465 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:30.878472 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:30.878704 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:30.879401 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:30.879423 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:30.882089 (+ 2666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:30.886147 (+ 4058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:30.886155 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:30.886171 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:30.886639 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:30.887052 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:30.887055 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:30.887058 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:30.887246 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:30.887908 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:30.887915 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:30.888258 (+ 343us) client_negotiation.cc:769] Sending connection context
0925 04:55:30.888552 (+ 294us) client_negotiation.cc:240] Negotiation successful
0925 04:55:30.888974 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":355,"threads_started":1}
I20260925 04:55:30.892071 14902 heartbeater.cc:347] Connected to a master server at 127.6.29.190:43689
I20260925 04:55:30.892400 14902 heartbeater.cc:464] Registering TS with master...
I20260925 04:55:30.893128 14902 heartbeater.cc:511] Master 127.6.29.190:43689 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.29.131@KRBTEST.COM; defaulting to no policy
I20260925 04:55:30.894661 14579 ts_manager.cc:194] Registered new tserver with Master: 4708ca219dc54da7a937934abc8f1e2b (127.6.29.130:42343)
I20260925 04:55:30.895494 14579 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.130@KRBTEST.COM'} at 127.6.29.130:36685
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab.
Entry for principal kudu/127.6.29.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab.
I20260925 04:55:30.923919 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.131:0
--local_ip_for_outbound_sockets=127.6.29.131
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:43689
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:34783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:55:31.248908 14907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:55:31.249274 14907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:55:31.249451 14907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:55:31.256868 14907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:55:31.257037 14907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:55:31.257259 14907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.131
I20260925 04:55:31.267128 14907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:34783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:43689
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.131
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:55:31.268530 14907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:55:31.269845 14907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:55:31.283171 14912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:31.283248 14913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:31.283411 14915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:55:31.284634 14907 server_base.cc:1034] running on GCE node
I20260925 04:55:31.285239 14907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:55:31.286168 14907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:55:31.287385 14907 hybrid_clock.cc:648] HybridClock initialized: now 1790312131287311 us; error 63 us; skew 500 ppm
I20260925 04:55:31.287747 14907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:55:31 dist-test-slave-xzjr krb5kdc[14534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312131, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:55:31.291115 14907 init.cc:377] Logged in from keytab as kudu/127.6.29.131@KRBTEST.COM (short username kudu)
I20260925 04:55:31.292680 14907 webserver.cc:460] Webserver started at http://127.6.29.131:35779/ using document root <none> and password file <none>
I20260925 04:55:31.293418 14907 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:31.293562 14907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:31.293907 14907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:55:31.297135 14907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-2/data/instance:
uuid: "ad5a3d053c8b4542bb07d5fb26b475cc"
format_stamp: "Formatted at 2026-09-25 04:55:31 on dist-test-slave-xzjr"
server_key: "51f386a404f5b5bd065a66b19703a1d7"
server_key_iv: "f864d0f2b58a867d65964cc68d398d53"
server_key_version: "encryptionkey@0"
I20260925 04:55:31.298010 14907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance:
uuid: "ad5a3d053c8b4542bb07d5fb26b475cc"
format_stamp: "Formatted at 2026-09-25 04:55:31 on dist-test-slave-xzjr"
server_key: "51f386a404f5b5bd065a66b19703a1d7"
server_key_iv: "f864d0f2b58a867d65964cc68d398d53"
server_key_version: "encryptionkey@0"
I20260925 04:55:31.303673 14907 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 04:55:31.307170 14922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:31.308596 14907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 04:55:31.308766 14907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-2/wal
uuid: "ad5a3d053c8b4542bb07d5fb26b475cc"
format_stamp: "Formatted at 2026-09-25 04:55:31 on dist-test-slave-xzjr"
server_key: "51f386a404f5b5bd065a66b19703a1d7"
server_key_iv: "f864d0f2b58a867d65964cc68d398d53"
server_key_version: "encryptionkey@0"
I20260925 04:55:31.308984 14907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:31.349210 14907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:55:31.353928 14907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:55:31.354241 14907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:55:31.355170 14907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:55:31.356509 14907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:55:31.356617 14907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:31.356765 14907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:55:31.356848 14907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:31.386060 14768 heartbeater.cc:503] Master 127.6.29.190:43689 was elected leader, sending a full tablet report...
I20260925 04:55:31.391291 14907 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.131:45779
I20260925 04:55:31.391314 15034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.131:45779 every 8 connection(s)
I20260925 04:55:31.392656 14907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
I20260925 04:55:31.395645 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 14907
I20260925 04:55:31.395846 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance
I20260925 04:55:31.396384 6262 external_mini_cluster.cc:1428] Setting key 7bd9ac8e2edf9f972c704c9bbd298bfd
Sep 25 04:55:31 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312131, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:55:31.413247 15037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:31.395085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:58399 (local address 127.6.29.190:43689)
0925 04:55:31.395510 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:31.395516 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:31.396905 (+ 1389us) server_negotiation.cc:407] Connection header received
0925 04:55:31.398932 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:31.398938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:31.399036 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:31.399159 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:31.400831 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:31.401668 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:31.403293 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:31.403705 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:31.403865 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:31.407128 (+ 3263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:31.407133 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:31.407137 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:31.407168 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:31.409873 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:31.410700 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:31.410705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:31.410708 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:31.410877 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:31.411339 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:31.411346 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:31.411350 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:31.411735 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:31.411920 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:31.412802 (+ 882us) server_negotiation.cc:299] Negotiation successful
0925 04:55:31.412981 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":210,"threads_started":1}
I20260925 04:55:31.414280 15038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:31.395855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.131:58399)
0925 04:55:31.396718 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:31.396769 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:31.398678 (+ 1909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:31.399423 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:31.399437 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:31.400028 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:31.400679 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:31.400699 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:31.401827 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:31.401832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:31.403174 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:31.403186 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:31.403935 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:31.403940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:31.404141 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:31.404752 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:31.404774 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:31.406914 (+ 2140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:31.410024 (+ 3110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:31.410030 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:31.410045 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:31.410536 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:31.411004 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:31.411010 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:31.411014 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:31.411221 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:31.411921 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:31.411932 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:31.412396 (+ 464us) client_negotiation.cc:769] Sending connection context
0925 04:55:31.412697 (+ 301us) client_negotiation.cc:240] Negotiation successful
0925 04:55:31.413170 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":317,"threads_started":1}
I20260925 04:55:31.415583 15035 heartbeater.cc:347] Connected to a master server at 127.6.29.190:43689
I20260925 04:55:31.415915 15035 heartbeater.cc:464] Registering TS with master...
I20260925 04:55:31.416574 15035 heartbeater.cc:511] Master 127.6.29.190:43689 requested a full tablet report, sending...
I20260925 04:55:31.417932 14579 ts_manager.cc:194] Registered new tserver with Master: ad5a3d053c8b4542bb07d5fb26b475cc (127.6.29.131:45779)
I20260925 04:55:31.418747 14579 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.131@KRBTEST.COM'} at 127.6.29.131:58399
I20260925 04:55:31.424429 6262 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.6.29.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.29.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.29.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.129.keytab.
Entry for principal oryx/127.6.29.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.129.keytab.
WARNING: no policy specified for oryx/127.6.29.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.29.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.29.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.130.keytab.
Entry for principal oryx/127.6.29.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.130.keytab.
WARNING: no policy specified for oryx/127.6.29.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.29.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.29.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.131.keytab.
Entry for principal oryx/127.6.29.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.131.keytab.
I20260925 04:55:31.567456 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 14640
Sep 25 04:55:31 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.129@KRBTEST.COM
I20260925 04:55:31.585019 15046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:31.568836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57048 (local address 127.6.29.129:39113)
0925 04:55:31.569644 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:31.569652 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:31.569679 (+ 27us) server_negotiation.cc:407] Connection header received
0925 04:55:31.569740 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:31.569746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:31.569961 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:31.570122 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:31.571413 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:31.572644 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:31.573938 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:31.574507 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:31.574612 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:31.578801 (+ 4189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:31.578814 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:31.578829 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:31.578873 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:31.582148 (+ 3275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:31.583036 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:31.583044 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:31.583046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:31.583208 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:31.583559 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:31.583563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:31.583565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:31.584013 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:31.584259 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:31.584517 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 04:55:31.584727 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":452,"threads_started":1}
I20260925 04:55:31.586362 14763 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:55:31.670490 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 14640
I20260925 04:55:31.687991 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 14774
Sep 25 04:55:31 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.130@KRBTEST.COM
I20260925 04:55:31.704622 15048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:31.688895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32820 (local address 127.6.29.130:42343)
0925 04:55:31.689476 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:31.689483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:31.689510 (+ 27us) server_negotiation.cc:407] Connection header received
0925 04:55:31.689576 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:31.689583 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:31.689796 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:31.689922 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:31.691086 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:31.692151 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:31.693405 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:31.693965 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:31.694061 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:31.698200 (+ 4139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:31.698212 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:31.698227 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:31.698261 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:31.701606 (+ 3345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:31.702464 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:31.702471 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:31.702479 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:31.702659 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:31.702989 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:31.702992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:31.702994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:31.703339 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:31.703636 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:31.704005 (+ 369us) server_negotiation.cc:299] Negotiation successful
0925 04:55:31.704209 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":321,"threads_started":1}
I20260925 04:55:31.706013 14897 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:55:31.791998 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 14774
I20260925 04:55:31.808079 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 14907
Sep 25 04:55:31 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.131@KRBTEST.COM
I20260925 04:55:31.824735 15050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:31.809175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33700 (local address 127.6.29.131:45779)
0925 04:55:31.809805 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:31.809812 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:31.809844 (+ 32us) server_negotiation.cc:407] Connection header received
0925 04:55:31.809905 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:31.809911 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:31.810147 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:31.810282 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:31.811552 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:31.812733 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:31.814098 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:31.814642 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:31.814741 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:31.819208 (+ 4467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:31.819220 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:31.819237 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:31.819271 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:31.822038 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:31.822695 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:31.822702 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:31.822704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:31.822821 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:31.823191 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:31.823196 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:31.823198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:31.823586 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:31.823915 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:31.824130 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 04:55:31.824332 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":298,"threads_started":1}
I20260925 04:55:31.826105 15030 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:55:31.909929 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 14907
I20260925 04:55:31.926074 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 14547
I20260925 04:55:31.927521 14609 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:55:32.030831 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 14547
I20260925 04:55:32.051437 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:43689
--webserver_interface=127.6.29.190
--webserver_port=42231
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:34783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:43689
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:55:32.394272 15053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:55:32.394585 15053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:55:32.394677 15053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:55:32.401921 15053 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:55:32.402030 15053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:55:32.402076 15053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:55:32.402127 15053 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:55:32.402170 15053 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:55:32.411957 15053 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:34783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:43689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:43689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=42231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:55:32.413277 15053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:55:32.415079 15053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:55:32.426756 15058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:32.426826 15059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:32.426854 15061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:55:32.427688 15053 server_base.cc:1034] running on GCE node
I20260925 04:55:32.428531 15053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:55:32.430014 15053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:55:32.431216 15053 hybrid_clock.cc:648] HybridClock initialized: now 1790312132431154 us; error 59 us; skew 500 ppm
I20260925 04:55:32.431586 15053 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:55:32 dist-test-slave-xzjr krb5kdc[14534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312132, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:55:32.435024 15053 init.cc:377] Logged in from keytab as kudu/127.6.29.190@KRBTEST.COM (short username kudu)
I20260925 04:55:32.436666 15053 webserver.cc:460] Webserver started at http://127.6.29.190:42231/ using document root <none> and password file <none>
I20260925 04:55:32.437381 15053 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:32.437529 15053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:32.443050 15053 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:55:32.445917 15068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:32.447238 15053 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:55:32.447414 15053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "950bf69bcda84cdeb862a86d57ff0e6e"
format_stamp: "Formatted at 2026-09-25 04:55:29 on dist-test-slave-xzjr"
server_key: "70deee2cece127e576e264cefbf50c44"
server_key_iv: "d74ebef282b814efd8c98cb7f7681cbd"
server_key_version: "encryptionkey@0"
I20260925 04:55:32.448465 15053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:32.478240 15053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:55:32.482870 15053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:55:32.483186 15053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:55:32.502782 15053 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:43689
I20260925 04:55:32.502806 15119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:43689 every 8 connection(s)
I20260925 04:55:32.504441 15053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:55:32.511605 15120 sys_catalog.cc:263] Verifying existing consensus state
I20260925 04:55:32.514248 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 15053
I20260925 04:55:32.515094 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.129:39113
--local_ip_for_outbound_sockets=127.6.29.129
--tserver_master_addrs=127.6.29.190:43689
--webserver_port=41977
--webserver_interface=127.6.29.129
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:34783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.29.129
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:55:32.515554 15120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e: Bootstrap starting.
I20260925 04:55:32.535151 15120 log.cc:824] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e: Log is configured to *not* fsync() on all Append() calls
I20260925 04:55:32.544366 15120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:55:32.545030 15120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e: Bootstrap complete.
I20260925 04:55:32.554332 15120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950bf69bcda84cdeb862a86d57ff0e6e" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43689 } }
I20260925 04:55:32.555151 15120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 950bf69bcda84cdeb862a86d57ff0e6e, State: Initialized, Role: FOLLOWER
I20260925 04:55:32.555845 15120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950bf69bcda84cdeb862a86d57ff0e6e" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43689 } }
I20260925 04:55:32.556085 15120 raft_consensus.cc:388] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:55:32.556200 15120 raft_consensus.cc:482] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:55:32.556386 15120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e [term 1 FOLLOWER]: Advancing to term 2
I20260925 04:55:32.560114 15120 raft_consensus.cc:504] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950bf69bcda84cdeb862a86d57ff0e6e" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43689 } }
I20260925 04:55:32.560595 15120 leader_election.cc:302] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 950bf69bcda84cdeb862a86d57ff0e6e; no voters:
I20260925 04:55:32.561240 15120 leader_election.cc:288] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 04:55:32.561371 15124 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e [term 2 FOLLOWER]: Leader election won for term 2
I20260925 04:55:32.562464 15124 raft_consensus.cc:686] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e [term 2 LEADER]: Becoming Leader. State: Replica: 950bf69bcda84cdeb862a86d57ff0e6e, State: Running, Role: LEADER
I20260925 04:55:32.563125 15124 consensus_queue.cc:237] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950bf69bcda84cdeb862a86d57ff0e6e" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43689 } }
I20260925 04:55:32.563503 15120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:55:32.566099 15126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 950bf69bcda84cdeb862a86d57ff0e6e. Latest consensus state: current_term: 2 leader_uuid: "950bf69bcda84cdeb862a86d57ff0e6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950bf69bcda84cdeb862a86d57ff0e6e" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43689 } } }
I20260925 04:55:32.566555 15126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e [sys.catalog]: This master's current role is: LEADER
I20260925 04:55:32.566797 15125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "950bf69bcda84cdeb862a86d57ff0e6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950bf69bcda84cdeb862a86d57ff0e6e" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43689 } } }
I20260925 04:55:32.567195 15125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e [sys.catalog]: This master's current role is: LEADER
I20260925 04:55:32.576874 15133 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:55:32.583788 15133 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:55:32.585425 15133 catalog_manager.cc:1259] Loaded cluster ID: fdb075eb35a54d6ba1a6abc94e2cb8a2
I20260925 04:55:32.585600 15133 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:55:32.589608 15133 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:55:32.591303 15133 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 950bf69bcda84cdeb862a86d57ff0e6e: Loaded TSK: 0
I20260925 04:55:32.592901 15133 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 04:55:32.868969 15122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:55:32.869321 15122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:55:32.869491 15122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:55:32.877327 15122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:55:32.877485 15122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:55:32.877677 15122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.129
I20260925 04:55:32.887997 15122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:34783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.129.keytab
--principal=oryx/127.6.29.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.129:39113
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.29.129
--webserver_port=41977
--tserver_master_addrs=127.6.29.190:43689
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.129
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:55:32.889410 15122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:55:32.890753 15122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:55:32.905236 15147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:32.905750 15146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:32.905817 15149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:55:32.909363 15122 server_base.cc:1034] running on GCE node
I20260925 04:55:32.910171 15122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:55:32.911448 15122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:55:32.912734 15122 hybrid_clock.cc:648] HybridClock initialized: now 1790312132912639 us; error 87 us; skew 500 ppm
I20260925 04:55:32.913146 15122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:55:32 dist-test-slave-xzjr krb5kdc[14534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312132, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.29.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:55:32.917335 15122 init.cc:377] Logged in from keytab as oryx/127.6.29.129@KRBTEST.COM (short username oryx)
I20260925 04:55:32.919114 15122 webserver.cc:460] Webserver started at http://127.6.29.129:41977/ using document root <none> and password file <none>
I20260925 04:55:32.920086 15122 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:32.920245 15122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:32.927441 15122 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.004s
I20260925 04:55:32.930881 15156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:32.932837 15122 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260925 04:55:32.933045 15122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-0/wal
uuid: "57326b8230174889be5a88abde39fdd8"
format_stamp: "Formatted at 2026-09-25 04:55:30 on dist-test-slave-xzjr"
server_key: "c5db73f2dfba2406d879f1b0c56b8448"
server_key_iv: "9960e994b6d33a7b13d00cd9cabccfb3"
server_key_version: "encryptionkey@0"
I20260925 04:55:32.934352 15122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:32.952250 15122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:55:32.956832 15122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:55:32.957137 15122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:55:32.958154 15122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:55:32.959503 15122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:55:32.959630 15122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:32.959798 15122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:55:32.959892 15122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260925 04:55:32.992415 15122 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.129:39113
I20260925 04:55:32.992511 15268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.129:39113 every 8 connection(s)
I20260925 04:55:32.993909 15122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
I20260925 04:55:32.998581 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 15122
I20260925 04:55:32.999182 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.130:42343
--local_ip_for_outbound_sockets=127.6.29.130
--tserver_master_addrs=127.6.29.190:43689
--webserver_port=46287
--webserver_interface=127.6.29.130
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:34783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.29.130
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 04:55:33 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.129@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:33 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.129@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:33.013312 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:32.996998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.129:47189)
0925 04:55:32.997856 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:32.997891 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:32.999737 (+ 1846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:33.001229 (+ 1492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:33.001242 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:33.001868 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:33.002415 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:33.002433 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:33.004105 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:33.004115 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:33.006009 (+ 1894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:33.006021 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:33.007534 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:33.007540 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:33.007747 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:33.008549 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:33.008572 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:33.012272 (+ 3700us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":369,"threads_started":1}
W20260925 04:55:33.014046 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:32.996871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:47189 (local address 127.6.29.190:43689)
0925 04:55:32.997683 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:32.997700 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:32.998036 (+ 336us) server_negotiation.cc:407] Connection header received
0925 04:55:33.000350 (+ 2314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:33.000368 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:33.000715 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:33.001082 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:33.002612 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:33.003943 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:33.006161 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:33.007297 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:33.007503 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:33.012766 (+ 5263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.129:47189: BlockingRecv error: recv got EOF from 127.6.29.129:47189 (error 108)
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":250,"threads_started":1}
W20260925 04:55:33.014213 15269 heartbeater.cc:644] Failed to heartbeat to 127.6.29.190:43689 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.29.190:43689: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Sep 25 04:55:33 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.129@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:33 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.129@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:33.024166 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:33.015141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.129:59587)
0925 04:55:33.015323 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:33.015347 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:33.015451 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:33.015870 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:33.015878 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:33.016198 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:33.016345 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:33.016354 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:33.017980 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:33.017984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:33.018925 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:33.018936 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:33.020058 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:33.020063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:33.020193 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:33.020890 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:33.020907 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:33.023954 (+ 3047us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 04:55:33.024446 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:33.015023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:59587 (local address 127.6.29.190:43689)
0925 04:55:33.015256 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:33.015272 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:33.015426 (+ 154us) server_negotiation.cc:407] Connection header received
0925 04:55:33.015569 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:33.015578 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:33.015676 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:33.015823 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:33.016497 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:33.017816 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:33.019075 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:33.019902 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:33.020059 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:33.024220 (+ 4161us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.129:59587: BlockingRecv error: recv got EOF from 127.6.29.129:59587 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 25 04:55:33 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.129@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:33 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.129@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:33.033110 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:33.025264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.129:34181)
0925 04:55:33.025416 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:33.025437 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:33.025542 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:33.025879 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:33.025884 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:33.026197 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:33.026342 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:33.026349 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:33.027351 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:33.027356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:33.028358 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:33.028369 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:33.029078 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:33.029083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:33.029306 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:33.029926 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:33.029943 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:33.032787 (+ 2844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 04:55:33.033277 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:33.025350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:34181 (local address 127.6.29.190:43689)
0925 04:55:33.025530 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:33.025535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:33.025553 (+ 18us) server_negotiation.cc:407] Connection header received
0925 04:55:33.025666 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:33.025671 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:33.025752 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:33.025857 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:33.026485 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:33.027203 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:33.028510 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:33.028928 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:33.029058 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:33.032995 (+ 3937us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.129:34181: BlockingRecv error: recv got EOF from 127.6.29.129:34181 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260925 04:55:33.033469 15269 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 04:55:33.337878 15273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:55:33.338168 15273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:55:33.338348 15273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:55:33.345717 15273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:55:33.345829 15273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:55:33.346037 15273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.130
I20260925 04:55:33.356093 15273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:34783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.130.keytab
--principal=oryx/127.6.29.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.130:42343
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.29.130
--webserver_port=46287
--tserver_master_addrs=127.6.29.190:43689
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.130
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:55:33.357448 15273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:55:33.358814 15273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:55:33.371878 15278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:33.372026 15279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:33.372376 15281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:55:33.372716 15273 server_base.cc:1034] running on GCE node
I20260925 04:55:33.373332 15273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:55:33.374264 15273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:55:33.375487 15273 hybrid_clock.cc:648] HybridClock initialized: now 1790312133375431 us; error 51 us; skew 500 ppm
I20260925 04:55:33.375878 15273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:55:33 dist-test-slave-xzjr krb5kdc[14534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312133, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.29.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:55:33.385391 15273 init.cc:377] Logged in from keytab as oryx/127.6.29.130@KRBTEST.COM (short username oryx)
I20260925 04:55:33.387156 15273 webserver.cc:460] Webserver started at http://127.6.29.130:46287/ using document root <none> and password file <none>
I20260925 04:55:33.388029 15273 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:33.388190 15273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:33.393906 15273 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 04:55:33.396855 15288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:33.398343 15273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 04:55:33.398535 15273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-1/wal
uuid: "4708ca219dc54da7a937934abc8f1e2b"
format_stamp: "Formatted at 2026-09-25 04:55:30 on dist-test-slave-xzjr"
server_key: "d52dc349b5eece226531bdfb540d770d"
server_key_iv: "ae8b7e323df176ae9a79732b821806d6"
server_key_version: "encryptionkey@0"
I20260925 04:55:33.399564 15273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:33.412122 15273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:55:33.416882 15273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:55:33.417205 15273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:55:33.418267 15273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:55:33.419703 15273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:55:33.419865 15273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260925 04:55:33.419991 15273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:55:33.420076 15273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:33.455050 15273 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.130:42343
I20260925 04:55:33.455145 15400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.130:42343 every 8 connection(s)
I20260925 04:55:33.456630 15273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
I20260925 04:55:33.464327 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 15273
I20260925 04:55:33.464983 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.131:45779
--local_ip_for_outbound_sockets=127.6.29.131
--tserver_master_addrs=127.6.29.190:43689
--webserver_port=35779
--webserver_interface=127.6.29.131
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:34783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.29.131
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 04:55:33 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.130@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:33 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.130@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
W20260925 04:55:33.472786 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:33.459218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:47873 (local address 127.6.29.190:43689)
0925 04:55:33.459391 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:33.459395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:33.460732 (+ 1337us) server_negotiation.cc:407] Connection header received
0925 04:55:33.462624 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:33.462630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:33.462722 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:33.462836 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:33.464635 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:33.465437 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:33.467010 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:33.467423 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:33.467566 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:33.472539 (+ 4973us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.130:47873: BlockingRecv error: recv got EOF from 127.6.29.130:47873 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 04:55:33.473814 15403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:33.459813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.130:47873)
0925 04:55:33.460571 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:33.460614 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:33.462342 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:33.463082 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:33.463091 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:33.463673 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:33.464411 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:33.464429 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:33.465603 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:33.465609 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:33.466857 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:33.466869 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:33.467671 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:33.467679 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:33.467975 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:33.468714 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:33.468737 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:33.472359 (+ 3622us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":311,"threads_started":1}
W20260925 04:55:33.474715 15401 heartbeater.cc:644] Failed to heartbeat to 127.6.29.190:43689 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.29.190:43689: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Sep 25 04:55:33 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.130@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:33 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.130@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:33.487913 15403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:33.476566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.130:58701)
0925 04:55:33.476831 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:33.476849 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:33.477017 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:33.477495 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:33.477500 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:33.477978 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:33.478157 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:33.478172 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:33.479664 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:33.479668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:33.480655 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:33.480668 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:33.481853 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:33.481861 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:33.482041 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:33.482740 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:33.482772 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:33.487483 (+ 4711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145}
W20260925 04:55:33.488094 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:33.476559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:58701 (local address 127.6.29.190:43689)
0925 04:55:33.476742 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:33.476748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:33.476918 (+ 170us) server_negotiation.cc:407] Connection header received
0925 04:55:33.477150 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:33.477156 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:33.477260 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:33.477321 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:33.478320 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:33.479509 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:33.480813 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:33.481609 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:33.481761 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:33.487856 (+ 6095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.130:58701: BlockingRecv error: recv got EOF from 127.6.29.130:58701 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 04:55:33 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.130@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:33 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.130@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:33.497237 15403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:33.488868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.130:48647)
0925 04:55:33.489051 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:33.489075 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:33.489212 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:33.489518 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:33.489523 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:33.489853 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:33.490008 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:33.490016 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:33.491036 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:33.491042 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:33.492070 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:33.492082 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:33.492823 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:33.492828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:33.493001 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:33.493643 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:33.493666 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:33.496898 (+ 3232us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 04:55:33.497609 15401 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 04:55:33.497783 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:33.489055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:48647 (local address 127.6.29.190:43689)
0925 04:55:33.489230 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:33.489235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:33.489255 (+ 20us) server_negotiation.cc:407] Connection header received
0925 04:55:33.489317 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:33.489322 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:33.489392 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:33.489494 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:33.490161 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:33.490893 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:33.492226 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:33.492669 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:33.492810 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:33.497149 (+ 4339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.130:48647: BlockingRecv error: recv got EOF from 127.6.29.130:48647 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 04:55:33.810920 15404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:55:33.811275 15404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:55:33.811460 15404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:55:33.818979 15404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:55:33.819159 15404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:55:33.819342 15404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.131
I20260925 04:55:33.829712 15404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:34783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/krb5kdc/oryx_127.6.29.131.keytab
--principal=oryx/127.6.29.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.131:45779
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.29.131
--webserver_port=35779
--tserver_master_addrs=127.6.29.190:43689
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.131
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:55:33.831143 15404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:55:33.832576 15404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:55:33.846199 15409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:33.846181 15410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:33.846735 15412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:55:33.846937 15404 server_base.cc:1034] running on GCE node
I20260925 04:55:33.847575 15404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:55:33.848579 15404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:55:33.849803 15404 hybrid_clock.cc:648] HybridClock initialized: now 1790312133849738 us; error 62 us; skew 500 ppm
I20260925 04:55:33.850164 15404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:55:33 dist-test-slave-xzjr krb5kdc[14534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312133, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.29.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:55:33.853637 15404 init.cc:377] Logged in from keytab as oryx/127.6.29.131@KRBTEST.COM (short username oryx)
I20260925 04:55:33.855219 15404 webserver.cc:460] Webserver started at http://127.6.29.131:35779/ using document root <none> and password file <none>
I20260925 04:55:33.856026 15404 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:33.856177 15404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:33.861580 15404 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 04:55:33.864418 15419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:33.865818 15404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:55:33.865994 15404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-2/wal
uuid: "ad5a3d053c8b4542bb07d5fb26b475cc"
format_stamp: "Formatted at 2026-09-25 04:55:31 on dist-test-slave-xzjr"
server_key: "51f386a404f5b5bd065a66b19703a1d7"
server_key_iv: "f864d0f2b58a867d65964cc68d398d53"
server_key_version: "encryptionkey@0"
I20260925 04:55:33.866981 15404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:33.898022 15404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:55:33.902456 15404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:55:33.902745 15404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:55:33.903872 15404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:55:33.905364 15404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:55:33.905454 15404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:33.905531 15404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:55:33.905583 15404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:33.939116 15404 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.131:45779
I20260925 04:55:33.939164 15531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.131:45779 every 8 connection(s)
I20260925 04:55:33.940667 15404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
I20260925 04:55:33.951067 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 15404
Sep 25 04:55:33 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.131@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:33 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.131@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
W20260925 04:55:33.957192 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:33.943122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:54857 (local address 127.6.29.190:43689)
0925 04:55:33.943357 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:33.943363 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:33.944685 (+ 1322us) server_negotiation.cc:407] Connection header received
0925 04:55:33.946496 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:33.946501 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:33.946566 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:33.946661 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:33.948269 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:33.948989 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:33.950545 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:33.950950 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:33.951097 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:33.956959 (+ 5862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.131:54857: BlockingRecv error: recv got EOF from 127.6.29.131:54857 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 04:55:33.958050 15534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:33.943701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.131:54857)
0925 04:55:33.944506 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:33.944550 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:33.946234 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:33.946882 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:33.946890 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:33.947422 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:33.948112 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:33.948131 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:33.949159 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:33.949163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:33.950417 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:33.950429 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:33.951213 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:33.951222 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:33.951444 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:33.952199 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:33.952229 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:33.956792 (+ 4563us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":297,"threads_started":1}
W20260925 04:55:33.958870 15532 heartbeater.cc:644] Failed to heartbeat to 127.6.29.190:43689 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.29.190:43689: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Sep 25 04:55:33 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.131@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:33 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.131@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:33.967998 15534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:33.959520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.131:44475)
0925 04:55:33.959718 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:33.959737 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:33.959864 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:33.960195 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:33.960198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:33.960489 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:33.960658 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:33.960665 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:33.962051 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:33.962058 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:33.962944 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:33.962955 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:33.963798 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:33.963805 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:33.963967 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:33.964520 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:33.964536 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:33.967680 (+ 3144us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 04:55:33.968179 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:33.959655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:44475 (local address 127.6.29.190:43689)
0925 04:55:33.959849 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:33.959855 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:33.959874 (+ 19us) server_negotiation.cc:407] Connection header received
0925 04:55:33.959982 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:33.959987 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:33.960069 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:33.960170 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:33.960799 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:33.961911 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:33.963086 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:33.963613 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:33.963740 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:33.967938 (+ 4198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.131:44475: BlockingRecv error: recv got EOF from 127.6.29.131:44475 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 04:55:33.969465 15536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:33.952260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49322 (local address 127.6.29.190:43689)
0925 04:55:33.953020 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:33.953026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:33.953044 (+ 18us) server_negotiation.cc:407] Connection header received
0925 04:55:33.953089 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:33.953093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:33.953155 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:33.953238 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:33.954918 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:33.955840 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:33.957679 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:33.958149 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:33.958283 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:33.962042 (+ 3759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:33.962051 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:33.962068 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:33.962103 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:33.965534 (+ 3431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:33.966238 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:33.966248 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:33.966258 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:33.966501 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:33.966927 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:33.966932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:33.966935 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:33.967558 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:33.968001 (+ 443us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:33.968611 (+ 610us) server_negotiation.cc:299] Negotiation successful
0925 04:55:33.969029 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":508,"threads_started":1}
Sep 25 04:55:33 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.131@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:33 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.131@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:33.977073 15534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:33.968941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.131:37529)
0925 04:55:33.969115 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:33.969134 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:33.969229 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:33.969521 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:33.969525 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:33.969819 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:33.969953 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:33.969961 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:33.971267 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:33.971274 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:33.972506 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:33.972517 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:33.973387 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:33.973393 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:33.973568 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:33.974051 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:33.974067 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:33.976768 (+ 2701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 04:55:33.977247 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:33.968990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:37529 (local address 127.6.29.190:43689)
0925 04:55:33.969203 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:33.969207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:33.969225 (+ 18us) server_negotiation.cc:407] Connection header received
0925 04:55:33.969333 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:33.969336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:33.969385 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:33.969466 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:33.970088 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:33.971123 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:33.972647 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:33.973252 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:33.973365 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:33.976992 (+ 3627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.131:37529: BlockingRecv error: recv got EOF from 127.6.29.131:37529 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
W20260925 04:55:33.977463 15532 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 04:55:34 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.129@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:34 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.129@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:34.043068 15537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:34.034054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.129:35151)
0925 04:55:34.034616 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:34.034631 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:34.034787 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:34.035135 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:34.035142 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:34.035631 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:34.035899 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:34.035910 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:34.036958 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:34.036963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:34.037933 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:34.037944 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:34.038754 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:34.038759 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:34.038937 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:34.039587 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:34.039611 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:34.042726 (+ 3115us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":308,"threads_started":1}
W20260925 04:55:34.043183 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:34.034216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:35151 (local address 127.6.29.190:43689)
0925 04:55:34.034468 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:34.034475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:34.034713 (+ 238us) server_negotiation.cc:407] Connection header received
0925 04:55:34.034914 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:34.034920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:34.034988 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:34.035092 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:34.036056 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:34.036802 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:34.038081 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:34.038549 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:34.038719 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:34.042948 (+ 4229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.129:35151: BlockingRecv error: recv got EOF from 127.6.29.129:35151 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 25 04:55:34 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.130@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:34 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.130@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:34.508003 15538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:34.498304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.130:49269)
0925 04:55:34.498926 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:34.498948 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:34.499132 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:34.499483 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:34.499488 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:34.500059 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:34.500245 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:34.500254 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:34.501257 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:34.501262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:34.502209 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:34.502220 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:34.503090 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:34.503097 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:34.503284 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:34.504065 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:34.504088 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:34.507666 (+ 3578us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":385,"threads_started":1}
W20260925 04:55:34.508167 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:34.498426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:49269 (local address 127.6.29.190:43689)
0925 04:55:34.498778 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:34.498785 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:34.499038 (+ 253us) server_negotiation.cc:407] Connection header received
0925 04:55:34.499284 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:34.499288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:34.499351 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:34.499452 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:34.500391 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:34.501102 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:34.502369 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:34.502867 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:34.503007 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:34.507928 (+ 4921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.130:49269: BlockingRecv error: recv got EOF from 127.6.29.130:49269 (error 108)
Metrics: {"server-negotiator.queue_time_us":187}
Sep 25 04:55:34 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.131@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:34 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.131@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:34.987855 15539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:34.978148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.131:41149)
0925 04:55:34.978721 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:34.978738 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:34.978865 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:34.979182 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:34.979185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:34.979650 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:34.979902 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:34.979914 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:34.981153 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:34.981158 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:34.982132 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:34.982145 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:34.982968 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:34.982974 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:34.983179 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:34.984236 (+ 1057us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:34.984261 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:34.987540 (+ 3279us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":321,"threads_started":1}
W20260925 04:55:34.988144 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:34.978200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:41149 (local address 127.6.29.190:43689)
0925 04:55:34.978375 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:34.978379 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:34.978830 (+ 451us) server_negotiation.cc:407] Connection header received
0925 04:55:34.978973 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:34.978977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:34.979042 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:34.979135 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:34.980043 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:34.980988 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:34.982297 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:34.982761 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:34.982871 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:34.987867 (+ 4996us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.131:41149: BlockingRecv error: recv got EOF from 127.6.29.131:41149 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 25 04:55:35 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.129@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:35 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.129@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:35.053580 15540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:35.044311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.129:40051)
0925 04:55:35.044913 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:35.044930 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:35.045058 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:35.045401 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:35.045405 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:35.045893 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:35.046041 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:35.046049 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:35.047329 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:35.047335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:35.048687 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:35.048701 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:35.049386 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:35.049390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:35.049505 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:35.050197 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:35.050214 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:35.053281 (+ 3067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":343,"threads_started":1}
W20260925 04:55:35.053730 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:35.044404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:40051 (local address 127.6.29.190:43689)
0925 04:55:35.044579 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:35.044583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:35.045017 (+ 434us) server_negotiation.cc:407] Connection header received
0925 04:55:35.045165 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:35.045169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:35.045258 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:35.045353 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:35.046217 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:35.047159 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:35.048814 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:35.049246 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:35.049348 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:35.053483 (+ 4135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.129:40051: BlockingRecv error: recv got EOF from 127.6.29.129:40051 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 04:55:35 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.130@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:35 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.130@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:35.517575 15541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:35.509239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.130:57603)
0925 04:55:35.509912 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:35.509929 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:35.510044 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:35.510345 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:35.510348 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:35.510762 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:35.510905 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:35.510914 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:35.511991 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:35.511995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:35.512892 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:35.512903 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:35.513628 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:35.513633 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:35.513741 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:35.514330 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:35.514347 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:35.517330 (+ 2983us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":469,"threads_started":1}
W20260925 04:55:35.517763 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:35.509300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:57603 (local address 127.6.29.190:43689)
0925 04:55:35.509475 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:35.509481 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:35.510025 (+ 544us) server_negotiation.cc:407] Connection header received
0925 04:55:35.510150 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:35.510154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:35.510218 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:35.510294 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:35.511069 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:35.511854 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:35.513028 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:35.513482 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:35.513585 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:35.517534 (+ 3949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.130:57603: BlockingRecv error: recv got EOF from 127.6.29.130:57603 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 04:55:35 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.131@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:35 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.131@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:35.998701 15542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:35.988866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.131:48329)
0925 04:55:35.989454 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:35.989470 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:35.989665 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:35.990019 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:35.990023 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:35.990453 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:35.990595 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:35.990611 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:35.991662 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:35.991667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:35.993132 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:35.993148 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:35.994105 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:35.994111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:35.994290 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:35.995171 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:35.995189 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:35.998432 (+ 3243us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":352,"threads_started":1}
W20260925 04:55:35.998946 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:35.989127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:48329 (local address 127.6.29.190:43689)
0925 04:55:35.989354 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:35.989361 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:35.989562 (+ 201us) server_negotiation.cc:407] Connection header received
0925 04:55:35.989817 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:35.989821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:35.989883 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:35.989989 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:35.990756 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:35.991510 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:35.993299 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:35.993947 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:35.994065 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:35.998663 (+ 4598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.131:48329: BlockingRecv error: recv got EOF from 127.6.29.131:48329 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 25 04:55:36 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.129@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:36 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.129@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:36.063643 15543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:36.054525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.129:60761)
0925 04:55:36.055078 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:36.055094 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:36.055265 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:36.055622 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:36.055627 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:36.056320 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:36.056604 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:36.056620 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:36.057697 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:36.057705 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:36.058689 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:36.058701 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:36.059386 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:36.059392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:36.059580 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:36.060260 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:36.060279 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:36.063332 (+ 3053us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":292,"threads_started":1}
W20260925 04:55:36.063862 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:36.054714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:60761 (local address 127.6.29.190:43689)
0925 04:55:36.054978 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:36.054985 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:36.055187 (+ 202us) server_negotiation.cc:407] Connection header received
0925 04:55:36.055397 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:36.055402 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:36.055475 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:36.055592 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:36.056772 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:36.057544 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:36.058831 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:36.059234 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:36.059363 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:36.063571 (+ 4208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.129:60761: BlockingRecv error: recv got EOF from 127.6.29.129:60761 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 25 04:55:36 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.130@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:36 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.130@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:36.527720 15544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:36.518641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.130:57445)
0925 04:55:36.519199 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:36.519218 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:36.519376 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:36.519807 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:36.519813 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:36.520170 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:36.520354 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:36.520363 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:36.521464 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:36.521469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:36.522485 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:36.522496 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:36.523411 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:36.523419 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:36.523576 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:36.524366 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:36.524386 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:36.527456 (+ 3070us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":285,"threads_started":1}
W20260925 04:55:36.528127 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:36.518660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:57445 (local address 127.6.29.190:43689)
0925 04:55:36.518872 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:36.518881 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:36.519310 (+ 429us) server_negotiation.cc:407] Connection header received
0925 04:55:36.519514 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:36.519520 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:36.519597 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:36.519712 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:36.520534 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:36.521336 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:36.522675 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:36.523193 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:36.523348 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:36.527840 (+ 4492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.130:57445: BlockingRecv error: recv got EOF from 127.6.29.130:57445 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 04:55:37 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.131@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:37 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.131@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:37.008510 15545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:36.999709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.131:37145)
0925 04:55:37.000201 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:37.000217 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:37.000334 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:37.000660 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:37.000663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:37.001004 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:37.001160 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:37.001168 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:37.002520 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:37.002526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:37.003646 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:37.003658 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:37.004385 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:37.004391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:37.004568 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:37.005197 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:37.005213 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:37.008117 (+ 2904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":287,"threads_started":1}
W20260925 04:55:37.008550 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:36.999904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:37145 (local address 127.6.29.190:43689)
0925 04:55:37.000136 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:37.000142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:37.000296 (+ 154us) server_negotiation.cc:407] Connection header received
0925 04:55:37.000431 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:37.000435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:37.000529 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:37.000613 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:37.001337 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:37.002374 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:37.003804 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:37.004233 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:37.004376 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:37.008332 (+ 3956us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.131:37145: BlockingRecv error: recv got EOF from 127.6.29.131:37145 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 25 04:55:37 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.129@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:37 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.129@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:37.073040 15546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:37.064562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.129:57611)
0925 04:55:37.065210 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:37.065234 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:37.065372 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:37.065682 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:37.065685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:37.066033 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:37.066188 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:37.066196 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:37.067261 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:37.067265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:37.068255 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:37.068267 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:37.068952 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:37.068956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:37.069107 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:37.069690 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:37.069706 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:37.072791 (+ 3085us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":331,"threads_started":1}
W20260925 04:55:37.073489 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:37.064684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:57611 (local address 127.6.29.190:43689)
0925 04:55:37.064950 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:37.064957 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:37.065326 (+ 369us) server_negotiation.cc:407] Connection header received
0925 04:55:37.065495 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:37.065499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:37.065559 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:37.065641 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:37.066347 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:37.067146 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:37.068380 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:37.068821 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:37.068920 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:37.073059 (+ 4139us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.129:57611: BlockingRecv error: recv got EOF from 127.6.29.129:57611 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 25 04:55:37 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.130@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:37 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.130@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:37.537182 15547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:37.528752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.130:48711)
0925 04:55:37.529266 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:37.529284 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:37.529411 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:37.529745 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:37.529748 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:37.530129 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:37.530282 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:37.530290 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:37.531258 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:37.531262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:37.532356 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:37.532368 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:37.533084 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:37.533089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:37.533224 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:37.533825 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:37.533842 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:37.536864 (+ 3022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":252,"threads_started":1}
W20260925 04:55:37.537379 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:37.528926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:48711 (local address 127.6.29.190:43689)
0925 04:55:37.529169 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:37.529177 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:37.529362 (+ 185us) server_negotiation.cc:407] Connection header received
0925 04:55:37.529531 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:37.529534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:37.529595 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:37.529700 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:37.530404 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:37.531127 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:37.532496 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:37.532920 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:37.533013 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:37.537117 (+ 4104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.130:48711: BlockingRecv error: recv got EOF from 127.6.29.130:48711 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 25 04:55:38 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.131@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:38 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.131@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:38.018076 15548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:38.009620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.131:33027)
0925 04:55:38.010323 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:38.010340 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:38.010499 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:38.010842 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:38.010846 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:38.011238 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:38.011379 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:38.011387 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:38.012487 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:38.012495 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:38.013450 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:38.013460 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:38.014175 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:38.014180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:38.014362 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:38.014974 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:38.014990 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:38.017835 (+ 2845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":347,"threads_started":1}
W20260925 04:55:38.018301 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:38.009749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:33027 (local address 127.6.29.190:43689)
0925 04:55:38.009956 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:38.009964 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:38.010429 (+ 465us) server_negotiation.cc:407] Connection header received
0925 04:55:38.010626 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:38.010634 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:38.010707 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:38.010817 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:38.011529 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:38.012352 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:38.013597 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:38.014018 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:38.014165 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:38.018031 (+ 3866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.131:33027: BlockingRecv error: recv got EOF from 127.6.29.131:33027 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 25 04:55:38 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.129@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:38 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.129@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:38.082329 15549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:38.074030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.129:52213)
0925 04:55:38.074513 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:38.074529 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:38.074700 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:38.075029 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:38.075034 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:38.075472 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:38.075632 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:38.075641 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:38.076745 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:38.076754 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:38.077739 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:38.077751 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:38.078461 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:38.078466 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:38.078644 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:38.079217 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:38.079233 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:38.081980 (+ 2747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":227,"threads_started":1}
W20260925 04:55:38.082408 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:38.074082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:52213 (local address 127.6.29.190:43689)
0925 04:55:38.074309 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:38.074318 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:38.074629 (+ 311us) server_negotiation.cc:407] Connection header received
0925 04:55:38.074823 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:38.074828 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:38.074893 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:38.074993 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:38.075811 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:38.076610 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:38.077882 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:38.078305 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:38.078453 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:38.082187 (+ 3734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.129:52213: BlockingRecv error: recv got EOF from 127.6.29.129:52213 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 25 04:55:38 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.130@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:38 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.130@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:38.553727 15550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:38.538194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.130:56805)
0925 04:55:38.538805 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:38.538827 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:38.538955 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:38.539446 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:38.539450 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:38.539984 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:38.540162 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:38.540172 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:38.541314 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:38.541318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:38.542319 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:38.542331 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:38.543026 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:38.543031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:38.543158 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:38.549839 (+ 6681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:38.549861 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:38.553342 (+ 3481us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":296,"threads_started":1}
W20260925 04:55:38.553838 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:38.538385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:56805 (local address 127.6.29.190:43689)
0925 04:55:38.538606 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:38.538612 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:38.538911 (+ 299us) server_negotiation.cc:407] Connection header received
0925 04:55:38.539100 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:38.539108 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:38.539181 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:38.539266 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:38.540300 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:38.541133 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:38.542452 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:38.542873 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:38.542966 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:38.553555 (+ 10589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.130:56805: BlockingRecv error: recv got EOF from 127.6.29.130:56805 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 25 04:55:39 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.131@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:39 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.131@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:39.027463 15551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:39.019223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.131:54439)
0925 04:55:39.019809 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:39.019831 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:39.019955 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:39.020323 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:39.020326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:39.020710 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:39.020866 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:39.020874 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:39.021850 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:39.021855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:39.022811 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:39.022822 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:39.023569 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:39.023574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:39.023721 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:39.024448 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:39.024465 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:39.027213 (+ 2748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":289,"threads_started":1}
W20260925 04:55:39.027853 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:39.019242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:54439 (local address 127.6.29.190:43689)
0925 04:55:39.019475 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:39.019485 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:39.019915 (+ 430us) server_negotiation.cc:407] Connection header received
0925 04:55:39.020108 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:39.020112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:39.020177 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:39.020266 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:39.020989 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:39.021709 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:39.022961 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:39.023401 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:39.023507 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:39.027500 (+ 3993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.131:54439: BlockingRecv error: recv got EOF from 127.6.29.131:54439 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 04:55:39 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.129@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:39 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.129@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:39.091454 15552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:39.083349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.129:46181)
0925 04:55:39.083888 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:39.083904 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:39.084030 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:39.084369 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:39.084373 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:39.084737 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:39.084898 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:39.084906 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:39.085913 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:39.085917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:39.086863 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:39.086873 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:39.087608 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:39.087613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:39.087744 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:39.088439 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:39.088457 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:39.091166 (+ 2709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":278,"threads_started":1}
W20260925 04:55:39.091800 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:39.083404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:46181 (local address 127.6.29.190:43689)
0925 04:55:39.083692 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:39.083698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:39.084000 (+ 302us) server_negotiation.cc:407] Connection header received
0925 04:55:39.084133 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:39.084137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:39.084212 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:39.084295 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:39.085050 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:39.085774 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:39.087029 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:39.087441 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:39.087558 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:39.091446 (+ 3888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.129:46181: BlockingRecv error: recv got EOF from 127.6.29.129:46181 (error 108)
Metrics: {"server-negotiator.queue_time_us":142}
Sep 25 04:55:39 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.130@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:39 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.130@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:39.563338 15553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:39.554797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.130:32975)
0925 04:55:39.555266 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:39.555284 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:39.555403 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:39.555709 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:39.555713 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:39.556141 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:39.556290 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:39.556298 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:39.557291 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:39.557295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:39.558205 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:39.558215 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:39.559208 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:39.559214 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:39.559389 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:39.560342 (+ 953us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:39.560370 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:39.563103 (+ 2733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":280,"threads_started":1}
W20260925 04:55:39.563669 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:39.554860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:32975 (local address 127.6.29.190:43689)
0925 04:55:39.555127 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:39.555133 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:39.555367 (+ 234us) server_negotiation.cc:407] Connection header received
0925 04:55:39.555507 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:39.555518 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:39.555585 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:39.555658 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:39.556437 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:39.557152 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:39.558343 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:39.559011 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:39.559137 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:39.563371 (+ 4234us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.130:32975: BlockingRecv error: recv got EOF from 127.6.29.130:32975 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 25 04:55:40 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.131@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:40 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.131@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:40.036762 15554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:40.028461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.131:43249)
0925 04:55:40.029024 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:40.029040 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:40.029156 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:40.029437 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:40.029440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:40.029822 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:40.029971 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:40.029979 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:40.031000 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:40.031004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:40.031968 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:40.031979 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:40.032719 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:40.032723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:40.032848 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:40.033489 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:40.033507 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:40.036448 (+ 2941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":342,"threads_started":1}
W20260925 04:55:40.037027 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:40.028678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:43249 (local address 127.6.29.190:43689)
0925 04:55:40.028857 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:40.028861 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:40.029120 (+ 259us) server_negotiation.cc:407] Connection header received
0925 04:55:40.029250 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:40.029253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:40.029311 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:40.029396 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:40.030122 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:40.030862 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:40.032117 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:40.032568 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:40.032670 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:40.036705 (+ 4035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.131:43249: BlockingRecv error: recv got EOF from 127.6.29.131:43249 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 25 04:55:40 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.129@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:40 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.129@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:40.101713 15555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:40.092461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.129:41423)
0925 04:55:40.093039 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:40.093055 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:40.093235 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:40.093632 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:40.093636 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:40.094182 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:40.094363 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:40.094373 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:40.095435 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:40.095440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:40.096442 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:40.096463 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:40.097183 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:40.097190 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:40.097359 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:40.097971 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:40.097990 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:40.101408 (+ 3418us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":261,"threads_started":1}
W20260925 04:55:40.101814 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:40.092615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:41423 (local address 127.6.29.190:43689)
0925 04:55:40.092814 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:40.092822 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:40.093155 (+ 333us) server_negotiation.cc:407] Connection header received
0925 04:55:40.093404 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:40.093408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:40.093486 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:40.093595 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:40.094528 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:40.095286 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:40.096607 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:40.097020 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:40.097164 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:40.101604 (+ 4440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.129:41423: BlockingRecv error: recv got EOF from 127.6.29.129:41423 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 04:55:40 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.130@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:40 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.130@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:40.573009 15556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:40.564282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.130:59335)
0925 04:55:40.564826 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:40.564842 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:40.564956 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:40.565281 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:40.565284 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:40.565701 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:40.565863 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:40.565871 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:40.566829 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:40.566833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:40.567744 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:40.567817 (+ 73us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:40.568512 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:40.568516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:40.568628 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:40.569253 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:40.569271 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:40.572711 (+ 3440us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":261,"threads_started":1}
W20260925 04:55:40.573184 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:40.564339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:59335 (local address 127.6.29.190:43689)
0925 04:55:40.564596 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:40.564602 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:40.564922 (+ 320us) server_negotiation.cc:407] Connection header received
0925 04:55:40.565068 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:40.565072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:40.565145 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:40.565227 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:40.565983 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:40.566705 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:40.567942 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:40.568342 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:40.568435 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:40.572918 (+ 4483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.130:59335: BlockingRecv error: recv got EOF from 127.6.29.130:59335 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 25 04:55:41 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.131@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:41 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.131@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:41.047133 15557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:41.037797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.131:39107)
0925 04:55:41.038418 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:41.038434 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:41.038602 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:41.038982 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:41.038985 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:41.039554 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:41.039736 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:41.039745 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:41.040868 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:41.040872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:41.041781 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:41.041792 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:41.042835 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:41.042844 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:41.043021 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:41.043742 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:41.043792 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:41.046881 (+ 3089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":308,"threads_started":1}
W20260925 04:55:41.047255 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:41.037929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:39107 (local address 127.6.29.190:43689)
0925 04:55:41.038102 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:41.038106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:41.038569 (+ 463us) server_negotiation.cc:407] Connection header received
0925 04:55:41.038731 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:41.038735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:41.038801 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:41.038891 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:41.039999 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:41.040746 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:41.041930 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:41.042617 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:41.042773 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:41.047086 (+ 4313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.131:39107: BlockingRecv error: recv got EOF from 127.6.29.131:39107 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 25 04:55:41 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.129@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:41 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.129@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:41.112205 15558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:41.102925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.129:41373)
0925 04:55:41.103510 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:41.103531 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:41.103714 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:41.104190 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:41.104196 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:41.104661 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:41.104876 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:41.104890 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:41.106096 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:41.106102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:41.107012 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:41.107023 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:41.107838 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:41.107845 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:41.108039 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:41.108816 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:41.108836 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:41.111844 (+ 3008us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":276,"threads_started":1}
W20260925 04:55:41.112318 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:41.103274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:41373 (local address 127.6.29.190:43689)
0925 04:55:41.103480 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:41.103484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:41.103664 (+ 180us) server_negotiation.cc:407] Connection header received
0925 04:55:41.103922 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:41.103928 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:41.104035 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:41.104131 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:41.105017 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:41.105943 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:41.107170 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:41.107610 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:41.107755 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:41.112060 (+ 4305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.129:41373: BlockingRecv error: recv got EOF from 127.6.29.129:41373 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 25 04:55:41 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.130@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:41 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.130@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:41.583191 15559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:41.574292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.130:55495)
0925 04:55:41.574807 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:41.574825 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:41.574952 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:41.575278 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:41.575281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:41.575653 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:41.575836 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:41.575846 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:41.577050 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:41.577054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:41.578108 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:41.578122 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:41.579168 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:41.579175 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:41.579374 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:41.580004 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:41.580022 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:41.582967 (+ 2945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":280,"threads_started":1}
W20260925 04:55:41.583415 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:41.574401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:55495 (local address 127.6.29.190:43689)
0925 04:55:41.574664 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:41.574670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:41.574914 (+ 244us) server_negotiation.cc:407] Connection header received
0925 04:55:41.575077 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:41.575081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:41.575145 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:41.575230 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:41.575965 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:41.576901 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:41.578268 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:41.578943 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:41.579098 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:41.583194 (+ 4096us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.130:55495: BlockingRecv error: recv got EOF from 127.6.29.130:55495 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
Sep 25 04:55:42 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.131@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:42 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.131@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:42.057191 15560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:42.048344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.131:51401)
0925 04:55:42.048870 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:42.048886 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:42.049041 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:42.049400 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:42.049405 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:42.049816 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:42.049960 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:42.049969 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:42.051114 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:42.051120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:42.052151 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:42.052163 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:42.053018 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:42.053024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:42.053186 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:42.053780 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:42.053796 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:42.056895 (+ 3099us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":277,"threads_started":1}
W20260925 04:55:42.057510 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:42.048437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:51401 (local address 127.6.29.190:43689)
0925 04:55:42.048660 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:42.048668 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:42.048971 (+ 303us) server_negotiation.cc:407] Connection header received
0925 04:55:42.049202 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:42.049207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:42.049268 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:42.049372 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:42.050104 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:42.050944 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:42.052291 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:42.052831 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:42.052961 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:42.057151 (+ 4190us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.131:51401: BlockingRecv error: recv got EOF from 127.6.29.131:51401 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 25 04:55:42 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.129@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:42 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.129@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:42.122390 15561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:42.113246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.129:57525)
0925 04:55:42.113807 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:42.113824 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:42.113938 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:42.114268 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:42.114272 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:42.114641 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:42.114801 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:42.114809 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:42.115886 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:42.115890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:42.116836 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:42.116847 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:42.117852 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:42.117859 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:42.118071 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:42.118838 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:42.118856 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:42.122133 (+ 3277us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":301,"threads_started":1}
W20260925 04:55:42.122675 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:42.113353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:57525 (local address 127.6.29.190:43689)
0925 04:55:42.113719 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:42.113728 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:42.113910 (+ 182us) server_negotiation.cc:407] Connection header received
0925 04:55:42.114026 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:42.114030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:42.114118 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:42.114208 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:42.114932 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:42.115725 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:42.116994 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:42.117673 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:42.117825 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:42.122395 (+ 4570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.129:57525: BlockingRecv error: recv got EOF from 127.6.29.129:57525 (error 108)
Metrics: {"server-negotiator.queue_time_us":223}
Sep 25 04:55:42 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.130@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:42 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.130@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:42.592809 15562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:42.584294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.130:36829)
0925 04:55:42.584854 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:42.584871 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:42.585027 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:42.585394 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:42.585398 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:42.585880 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:42.586041 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:42.586050 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:42.587093 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:42.587099 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:42.588194 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:42.588207 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:42.588867 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:42.588872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:42.588992 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:42.589633 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:42.589652 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:42.592546 (+ 2894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":299,"threads_started":1}
W20260925 04:55:42.593004 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:42.584376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:36829 (local address 127.6.29.190:43689)
0925 04:55:42.584577 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:42.584583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:42.584957 (+ 374us) server_negotiation.cc:407] Connection header received
0925 04:55:42.585171 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:42.585176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:42.585255 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:42.585350 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:42.586216 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:42.586950 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:42.588323 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:42.588738 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:42.588832 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:42.592754 (+ 3922us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.130:36829: BlockingRecv error: recv got EOF from 127.6.29.130:36829 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 25 04:55:43 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.131@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:43 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.131@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:43.066923 15563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:43.058153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.131:60695)
0925 04:55:43.058747 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:43.058765 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:43.058880 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:43.059190 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:43.059193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:43.059559 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:43.059727 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:43.059735 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:43.060805 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:43.060811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:43.061775 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:43.061789 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:43.062451 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:43.062456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:43.062581 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:43.063253 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:43.063276 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:43.066658 (+ 3382us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":339,"threads_started":1}
W20260925 04:55:43.067080 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:43.058226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:60695 (local address 127.6.29.190:43689)
0925 04:55:43.058451 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:43.058456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:43.058841 (+ 385us) server_negotiation.cc:407] Connection header received
0925 04:55:43.058986 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:43.058990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:43.059064 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:43.059149 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:43.059925 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:43.060661 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:43.061901 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:43.062300 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:43.062390 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:43.066872 (+ 4482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.131:60695: BlockingRecv error: recv got EOF from 127.6.29.131:60695 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 25 04:55:43 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.129@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:43 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.129@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:43.132444 15564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:43.123422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.129:56853)
0925 04:55:43.124036 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:43.124064 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:43.124192 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:43.124583 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:43.124586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:43.124958 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:43.125127 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:43.125137 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:43.126552 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:43.126556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:43.127673 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:43.127687 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:43.128466 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:43.128470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:43.128591 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:43.129217 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:43.129234 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:43.132153 (+ 2919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":386,"threads_started":1}
W20260925 04:55:43.132638 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:43.123579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:56853 (local address 127.6.29.190:43689)
0925 04:55:43.123856 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:43.123861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:43.124175 (+ 314us) server_negotiation.cc:407] Connection header received
0925 04:55:43.124362 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:43.124366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:43.124438 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:43.124541 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:43.125401 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:43.126384 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:43.127866 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:43.128327 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:43.128431 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:43.132367 (+ 3936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.129:56853: BlockingRecv error: recv got EOF from 127.6.29.129:56853 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 25 04:55:43 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.130@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:43 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.130@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:43.606856 15565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:43.593942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.130:36459)
0925 04:55:43.594478 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:43.594495 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:43.594613 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:43.595070 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:43.595074 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:43.595603 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:43.595824 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:43.595838 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:43.597073 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:43.597077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:43.597996 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:43.598007 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:43.598742 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:43.598746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:43.598868 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:43.603418 (+ 4550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:43.603439 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:43.606608 (+ 3169us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":320,"threads_started":1}
W20260925 04:55:43.607262 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:43.594125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:36459 (local address 127.6.29.190:43689)
0925 04:55:43.594348 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:43.594355 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:43.594600 (+ 245us) server_negotiation.cc:407] Connection header received
0925 04:55:43.594805 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:43.594809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:43.594909 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:43.595026 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:43.596014 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:43.596940 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:43.598146 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:43.598590 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:43.598691 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:43.606986 (+ 8295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.130:36459: BlockingRecv error: recv got EOF from 127.6.29.130:36459 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 25 04:55:44 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.131@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:44 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.131@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:44.076695 15566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:44.068086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.131:44087)
0925 04:55:44.068644 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:44.068662 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:44.068774 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:44.069077 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:44.069080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:44.069423 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:44.069588 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:44.069598 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:44.070709 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:44.070714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:44.071714 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:44.071726 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:44.072455 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:44.072459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:44.072599 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:44.073215 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:44.073232 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:44.076258 (+ 3026us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":320,"threads_started":1}
W20260925 04:55:44.076776 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:44.068185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:44087 (local address 127.6.29.190:43689)
0925 04:55:44.068402 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:44.068406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:44.068741 (+ 335us) server_negotiation.cc:407] Connection header received
0925 04:55:44.068878 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:44.068881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:44.068943 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:44.069025 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:44.069769 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:44.070572 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:44.071925 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:44.072317 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:44.072412 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:44.076497 (+ 4085us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.131:44087: BlockingRecv error: recv got EOF from 127.6.29.131:44087 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Sep 25 04:55:44 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.129@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:44 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.129@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:44.142504 15567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:44.133401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.129:59277)
0925 04:55:44.133965 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:44.133979 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:44.134155 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:44.134502 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:44.134506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:44.134941 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:44.135109 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:44.135133 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:44.136299 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:44.136305 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:44.137215 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:44.137226 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:44.138038 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:44.138052 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:44.138246 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:44.139046 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:44.139064 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:44.142258 (+ 3194us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":332,"threads_started":1}
W20260925 04:55:44.142791 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:44.133496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:59277 (local address 127.6.29.190:43689)
0925 04:55:44.133784 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:44.133793 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:44.134075 (+ 282us) server_negotiation.cc:407] Connection header received
0925 04:55:44.134300 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:44.134305 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:44.134366 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:44.134473 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:44.135271 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:44.136161 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:44.137371 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:44.137836 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:44.137962 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:44.142493 (+ 4531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.129:59277: BlockingRecv error: recv got EOF from 127.6.29.129:59277 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 04:55:44 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.130@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:44 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.130@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:44.616590 15568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:44.607963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.130:32797)
0925 04:55:44.608555 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:44.608572 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:44.608763 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:44.609125 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:44.609142 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:44.609566 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:44.609732 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:44.609740 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:44.610755 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:44.610763 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:44.611676 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:44.611687 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:44.612447 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:44.612453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:44.612640 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:44.613271 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:44.613289 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:44.616204 (+ 2915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":366,"threads_started":1}
W20260925 04:55:44.616681 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:44.608063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:32797 (local address 127.6.29.190:43689)
0925 04:55:44.608309 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:44.608317 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:44.608682 (+ 365us) server_negotiation.cc:407] Connection header received
0925 04:55:44.608904 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:44.608909 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:44.608983 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:44.609091 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:44.609884 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:44.610602 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:44.611883 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:44.612289 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:44.612432 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:44.616424 (+ 3992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.130:32797: BlockingRecv error: recv got EOF from 127.6.29.130:32797 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 25 04:55:45 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.131@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:45 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.131@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:45.087121 15569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:45.077845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.131:46281)
0925 04:55:45.078414 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:45.078430 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:45.078617 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:45.078971 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:45.078997 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:45.079419 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:45.079598 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:45.079607 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:45.080696 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:45.080702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:45.081605 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:45.081616 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:45.082330 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:45.082337 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:45.082505 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:45.083229 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:45.083248 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:45.086813 (+ 3565us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":310,"threads_started":1}
W20260925 04:55:45.087261 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:45.077929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:46281 (local address 127.6.29.190:43689)
0925 04:55:45.078205 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:45.078214 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:45.078560 (+ 346us) server_negotiation.cc:407] Connection header received
0925 04:55:45.078763 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:45.078767 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:45.078828 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:45.078935 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:45.079749 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:45.080549 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:45.081753 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:45.082158 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:45.082290 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:45.087017 (+ 4727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.131:46281: BlockingRecv error: recv got EOF from 127.6.29.131:46281 (error 108)
Metrics: {"server-negotiator.queue_time_us":117}
Sep 25 04:55:45 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.129@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:45 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.129@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:45.153296 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:45.143570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.129:40241)
0925 04:55:45.144173 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:45.144190 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:45.144315 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:45.144712 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:45.144716 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:45.145124 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:45.145287 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:45.145296 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:45.146527 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:45.146531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:45.147458 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:45.147519 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:45.148894 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:45.148900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:45.149084 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:45.149872 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:45.149890 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:45.153004 (+ 3114us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":347,"threads_started":1}
W20260925 04:55:45.153571 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:45.143644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:40241 (local address 127.6.29.190:43689)
0925 04:55:45.143884 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:45.143891 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:45.144308 (+ 417us) server_negotiation.cc:407] Connection header received
0925 04:55:45.144424 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:45.144428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:45.144494 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:45.144612 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:45.145520 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:45.146363 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:45.147739 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:45.148710 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:45.148870 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:45.153266 (+ 4396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.129:40241: BlockingRecv error: recv got EOF from 127.6.29.129:40241 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 25 04:55:45 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.130@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:45 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.130@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:45.625932 15571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:45.617606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.130:58131)
0925 04:55:45.618083 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:45.618100 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:45.618216 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:45.618542 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:45.618545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:45.618938 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:45.619094 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:45.619102 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:45.620176 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:45.620180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:45.621101 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:45.621114 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:45.621948 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:45.621954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:45.622141 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:45.622886 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:45.622903 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:45.625704 (+ 2801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":281,"threads_started":1}
W20260925 04:55:45.626121 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:45.617693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:58131 (local address 127.6.29.190:43689)
0925 04:55:45.617941 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:45.617947 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:45.618186 (+ 239us) server_negotiation.cc:407] Connection header received
0925 04:55:45.618327 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:45.618331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:45.618407 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:45.618489 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:45.619222 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:45.620047 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:45.621247 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:45.621798 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:45.621937 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:45.625880 (+ 3943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.130:58131: BlockingRecv error: recv got EOF from 127.6.29.130:58131 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 25 04:55:46 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.131@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:46 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.131@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:46.097147 15572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:46.088157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.131:53293)
0925 04:55:46.088726 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:46.088741 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:46.088910 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:46.089294 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:46.089298 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:46.089698 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:46.089851 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:46.089859 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:46.090917 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:46.090924 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:46.091895 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:46.091906 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:46.092822 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:46.092829 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:46.093023 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:46.093854 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:46.093885 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:46.096917 (+ 3032us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":310,"threads_started":1}
W20260925 04:55:46.097452 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:46.088291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:53293 (local address 127.6.29.190:43689)
0925 04:55:46.088595 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:46.088601 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:46.088840 (+ 239us) server_negotiation.cc:407] Connection header received
0925 04:55:46.089067 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:46.089075 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:46.089143 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:46.089252 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:46.090005 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:46.090764 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:46.092098 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:46.092645 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:46.092780 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:46.097145 (+ 4365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.131:53293: BlockingRecv error: recv got EOF from 127.6.29.131:53293 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 25 04:55:46 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.129@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:46 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.129@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:46.162988 15573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:46.154377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.129:52413)
0925 04:55:46.155014 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:46.155031 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:46.155151 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:46.155540 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:46.155544 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:46.156101 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:46.156263 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:46.156271 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:46.157287 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:46.157291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:46.158230 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:46.158241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:46.159056 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:46.159070 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:46.159190 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:46.159839 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:46.159856 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:46.162752 (+ 2896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":366,"threads_started":1}
W20260925 04:55:46.163277 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:46.154556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:52413 (local address 127.6.29.190:43689)
0925 04:55:46.154747 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:46.154754 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:46.155131 (+ 377us) server_negotiation.cc:407] Connection header received
0925 04:55:46.155271 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:46.155276 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:46.155358 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:46.155479 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:46.156393 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:46.157155 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:46.158381 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:46.158825 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:46.158936 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:46.162986 (+ 4050us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.129:52413: BlockingRecv error: recv got EOF from 127.6.29.129:52413 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 04:55:46 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.130@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:46 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.130@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:46.635646 15574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:46.626901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.130:44923)
0925 04:55:46.627473 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:46.627489 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:46.627669 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:46.628051 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:46.628056 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:46.628453 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:46.628605 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:46.628623 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:46.629635 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:46.629640 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:46.630551 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:46.630561 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:46.631313 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:46.631320 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:46.631491 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:46.632199 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:46.632216 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:46.635290 (+ 3074us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":275,"threads_started":1}
W20260925 04:55:46.635730 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:46.627187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:44923 (local address 127.6.29.190:43689)
0925 04:55:46.627408 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:46.627414 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:46.627578 (+ 164us) server_negotiation.cc:407] Connection header received
0925 04:55:46.627839 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:46.627843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:46.627915 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:46.628021 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:46.628763 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:46.629487 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:46.630711 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:46.631133 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:46.631277 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:46.635494 (+ 4217us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.130:44923: BlockingRecv error: recv got EOF from 127.6.29.130:44923 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 25 04:55:47 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.131@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:47 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.131@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:47.107425 15575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:47.098312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.131:55315)
0925 04:55:47.098906 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:47.098923 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:47.099114 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:47.099461 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:47.099465 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:47.099918 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:47.100082 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:47.100091 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:47.101402 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:47.101409 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:47.102433 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:47.102445 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:47.103228 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:47.103235 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:47.103399 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:47.104061 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:47.104092 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:47.107080 (+ 2988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":329,"threads_started":1}
W20260925 04:55:47.107590 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:47.098444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:55315 (local address 127.6.29.190:43689)
0925 04:55:47.098745 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:47.098751 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:47.099030 (+ 279us) server_negotiation.cc:407] Connection header received
0925 04:55:47.099253 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:47.099263 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:47.099324 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:47.099430 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:47.100232 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:47.101271 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:47.102595 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:47.103011 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:47.103147 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:47.107291 (+ 4144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.131:55315: BlockingRecv error: recv got EOF from 127.6.29.131:55315 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Sep 25 04:55:47 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.129@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:47 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.129@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:47.172212 15576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:47.164044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.129:47799)
0925 04:55:47.164593 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:47.164651 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:47.164782 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:47.165028 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:47.165031 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:47.165377 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:47.165552 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:47.165560 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:47.166564 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:47.166567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:47.167440 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:47.167449 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:47.168129 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:47.168133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:47.168250 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:47.168953 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:47.168970 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:47.171962 (+ 2992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":279,"threads_started":1}
W20260925 04:55:47.172449 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:47.164050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:47799 (local address 127.6.29.190:43689)
0925 04:55:47.164257 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:47.164261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:47.164736 (+ 475us) server_negotiation.cc:407] Connection header received
0925 04:55:47.164878 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:47.164882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:47.164941 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:47.165012 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:47.165681 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:47.166419 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:47.167570 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:47.168006 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:47.168106 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:47.172184 (+ 4078us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.129:47799: BlockingRecv error: recv got EOF from 127.6.29.129:47799 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Sep 25 04:55:47 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.130@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:47 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.130@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:47.645356 15577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:47.636708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.130:44883)
0925 04:55:47.637294 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:47.637312 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:47.637429 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:47.637802 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:47.637807 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:47.638202 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:47.638370 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:47.638379 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:47.639489 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:47.639494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:47.640494 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:47.640506 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:47.641225 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:47.641231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:47.641406 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:47.642056 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:47.642073 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:47.645128 (+ 3055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":325,"threads_started":1}
W20260925 04:55:47.645566 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:47.636865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:44883 (local address 127.6.29.190:43689)
0925 04:55:47.637088 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:47.637094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:47.637397 (+ 303us) server_negotiation.cc:407] Connection header received
0925 04:55:47.637581 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:47.637585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:47.637661 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:47.637769 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:47.638543 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:47.639326 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:47.640661 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:47.641067 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:47.641203 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:47.645361 (+ 4158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.130:44883: BlockingRecv error: recv got EOF from 127.6.29.130:44883 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 25 04:55:48 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.131@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:48 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.131@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:48.123709 15578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:48.108611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.131:57919)
0925 04:55:48.109333 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:48.109349 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:48.109489 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:48.109832 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:48.109836 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:48.110186 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:48.110352 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:48.110360 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:48.111302 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:48.111305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:48.112250 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:48.112261 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:48.112899 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:48.112903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:48.113019 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:48.120368 (+ 7349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:48.120388 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:48.123467 (+ 3079us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":488,"threads_started":1}
W20260925 04:55:48.124027 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:48.108706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:57919 (local address 127.6.29.190:43689)
0925 04:55:48.108932 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:48.108936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:48.109407 (+ 471us) server_negotiation.cc:407] Connection header received
0925 04:55:48.109627 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:48.109632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:48.109705 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:48.109784 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:48.110470 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:48.111178 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:48.112375 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:48.112778 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:48.112873 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:48.123731 (+ 10858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.131:57919: BlockingRecv error: recv got EOF from 127.6.29.131:57919 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 25 04:55:48 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.129@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:48 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.129@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:48.186147 15579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:48.173497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.129:33975)
0925 04:55:48.173995 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:48.174019 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:48.174176 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:48.174545 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:48.174550 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:48.174973 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:48.175171 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:48.175181 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:48.177095 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:48.177101 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:48.178738 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:48.178755 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:48.179630 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:48.179636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:48.179928 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:48.181031 (+ 1103us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:48.181057 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:48.185761 (+ 4704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":275,"threads_started":1}
W20260925 04:55:48.186208 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:48.173548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:33975 (local address 127.6.29.190:43689)
0925 04:55:48.173749 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:48.173757 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:48.174109 (+ 352us) server_negotiation.cc:407] Connection header received
0925 04:55:48.174320 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:48.174325 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:48.174409 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:48.174509 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:48.175333 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:48.176927 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:48.178921 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:48.179453 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:48.179622 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:48.185981 (+ 6359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.129:33975: BlockingRecv error: recv got EOF from 127.6.29.129:33975 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 04:55:48 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.130@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 04:55:48 dist-test-slave-xzjr krb5kdc[14534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.29.130@KRBTEST.COM for oryx/127.6.29.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 04:55:48.660971 15580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:48.646595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43689 (local address 127.6.29.130:41275)
0925 04:55:48.647161 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:48.647177 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:48.647339 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:48.647712 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:48.647717 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:48.648168 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:48.648358 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:48.648367 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:48.649359 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:48.649365 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:48.650277 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:48.650288 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:48.651033 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:48.651039 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:48.651223 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:48.657652 (+ 6429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:48.657674 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:48.660691 (+ 3017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.29.190:43689: Server oryx/127.6.29.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":295,"threads_started":1}
W20260925 04:55:48.661276 15271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:48.646742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:41275 (local address 127.6.29.190:43689)
0925 04:55:48.646982 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:48.646988 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:48.647271 (+ 283us) server_negotiation.cc:407] Connection header received
0925 04:55:48.647475 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:48.647480 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:48.647549 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:48.647673 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:48.648504 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:48.649214 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:48.650414 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:48.650879 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:48.651044 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:48.660972 (+ 9928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.29.130:41275: BlockingRecv error: recv got EOF from 127.6.29.130:41275 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
I20260925 04:55:48.957243 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 15122
I20260925 04:55:48.969766 15582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:48.958398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41132 (local address 127.6.29.129:39113)
0925 04:55:48.959003 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:48.959011 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:48.959037 (+ 26us) server_negotiation.cc:407] Connection header received
0925 04:55:48.959097 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:48.959104 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:48.959339 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:48.959498 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:48.961046 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:48.962019 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:48.963216 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:48.963857 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:48.963977 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:48.966488 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:48.966497 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:48.966511 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:48.966544 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:48.969159 (+ 2615us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.29.129@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 04:55:48.969489 (+ 330us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41132: Request ticket server kudu/127.6.29.129@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":317,"threads_started":1}
W20260925 04:55:48.969877 15582 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41132: Request ticket server kudu/127.6.29.129@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 04:55:48.969866 6262 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 15122: Not authorized: Client connection negotiation failed: client connection to 127.6.29.129:39113: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.29.129@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 04:55:48.970156 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 15122
I20260925 04:55:48.987272 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 15273
I20260925 04:55:48.999752 15583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:48.988464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42900 (local address 127.6.29.130:42343)
0925 04:55:48.989052 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:48.989061 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:48.989092 (+ 31us) server_negotiation.cc:407] Connection header received
0925 04:55:48.989151 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:48.989157 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:48.989376 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:48.989525 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:48.990823 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:48.991995 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:48.993224 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:48.993874 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:48.993985 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:48.996477 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:48.996486 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:48.996501 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:48.996538 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:48.999040 (+ 2502us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.29.130@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 04:55:48.999404 (+ 364us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42900: Request ticket server kudu/127.6.29.130@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":296,"threads_started":1}
W20260925 04:55:48.999814 6262 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 15273: Not authorized: Client connection negotiation failed: client connection to 127.6.29.130:42343: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.29.130@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 04:55:48.999933 15583 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42900: Request ticket server kudu/127.6.29.130@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 04:55:48.999984 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 15273
I20260925 04:55:49.013441 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 15404
I20260925 04:55:49.025144 15584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:49.014429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48510 (local address 127.6.29.131:45779)
0925 04:55:49.014999 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:49.015006 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:49.015032 (+ 26us) server_negotiation.cc:407] Connection header received
0925 04:55:49.015097 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:49.015104 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:49.015320 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:49.015469 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:49.016897 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:49.017920 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:49.019091 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:49.019630 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:49.019726 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:49.022175 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:49.022183 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:49.022198 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:49.022244 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:49.024642 (+ 2398us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.29.131@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 04:55:49.024900 (+ 258us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48510: Request ticket server kudu/127.6.29.131@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":317,"threads_started":1}
W20260925 04:55:49.025272 15584 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48510: Request ticket server kudu/127.6.29.131@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 04:55:49.025417 6262 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 15404: Not authorized: Client connection negotiation failed: client connection to 127.6.29.131:45779: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.29.131@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 04:55:49.025585 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 15404
I20260925 04:55:49.042294 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 15053
I20260925 04:55:49.043718 15115 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:55:49.158985 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 15053
2026-09-25T04:55:49Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21211 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T04:55:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:55:49Z Disabled control of system clock
I20260925 04:55:49.236080 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:35249
--webserver_interface=127.6.29.190
--webserver_port=0
--builtin_ntp_servers=127.6.29.148:39641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:35249
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 04:55:49.577124 15589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:55:49.577458 15589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:55:49.577538 15589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:55:49.584847 15589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:55:49.584966 15589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:55:49.585003 15589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:55:49.585052 15589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:55:49.585094 15589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:55:49.594321 15589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:39641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:35249
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:35249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:55:49.595618 15589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:55:49.597458 15589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:55:49.609520 15594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:49.609588 15597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:49.609645 15595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:55:49.610244 15589 server_base.cc:1034] running on GCE node
I20260925 04:55:49.611055 15589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:55:49.612579 15589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:55:49.613816 15589 hybrid_clock.cc:648] HybridClock initialized: now 1790312149613775 us; error 48 us; skew 500 ppm
I20260925 04:55:49.614177 15589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 04:55:49.616647 15589 webserver.cc:460] Webserver started at http://127.6.29.190:45795/ using document root <none> and password file <none>
I20260925 04:55:49.617412 15589 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:49.617551 15589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:49.617918 15589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:55:49.621076 15589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "76350eb59ef944948e16327cd1a3081f"
format_stamp: "Formatted at 2026-09-25 04:55:49 on dist-test-slave-xzjr"
server_key: "a33340bce1eb6d0147e3e77de4d50a14"
server_key_iv: "b2f75b2913940bea6ca839b08e4a213a"
server_key_version: "encryptionkey@0"
I20260925 04:55:49.621874 15589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "76350eb59ef944948e16327cd1a3081f"
format_stamp: "Formatted at 2026-09-25 04:55:49 on dist-test-slave-xzjr"
server_key: "a33340bce1eb6d0147e3e77de4d50a14"
server_key_iv: "b2f75b2913940bea6ca839b08e4a213a"
server_key_version: "encryptionkey@0"
I20260925 04:55:49.627135 15589 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 04:55:49.630548 15603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:49.631925 15589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 04:55:49.632107 15589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "76350eb59ef944948e16327cd1a3081f"
format_stamp: "Formatted at 2026-09-25 04:55:49 on dist-test-slave-xzjr"
server_key: "a33340bce1eb6d0147e3e77de4d50a14"
server_key_iv: "b2f75b2913940bea6ca839b08e4a213a"
server_key_version: "encryptionkey@0"
I20260925 04:55:49.632320 15589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:49.657891 15589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:55:49.658823 15589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:55:49.659130 15589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:55:49.677798 15589 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:35249
I20260925 04:55:49.677821 15654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:35249 every 8 connection(s)
I20260925 04:55:49.679251 15589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:55:49.683290 15655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:55:49.688066 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 15589
I20260925 04:55:49.688292 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 04:55:49.688947 6262 external_mini_cluster.cc:1428] Setting key 89196a96cbc1472b6dc9cd57ceff203e
I20260925 04:55:49.696000 15655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76350eb59ef944948e16327cd1a3081f: Bootstrap starting.
I20260925 04:55:49.699935 15655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76350eb59ef944948e16327cd1a3081f: Neither blocks nor log segments found. Creating new log.
I20260925 04:55:49.701313 15655 log.cc:824] T 00000000000000000000000000000000 P 76350eb59ef944948e16327cd1a3081f: Log is configured to *not* fsync() on all Append() calls
I20260925 04:55:49.705068 15655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76350eb59ef944948e16327cd1a3081f: No bootstrap required, opened a new log
I20260925 04:55:49.705641 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:49.691224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37694 (local address 127.6.29.190:35249)
0925 04:55:49.692494 (+ 1270us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:49.692509 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:49.692566 (+ 57us) server_negotiation.cc:407] Connection header received
0925 04:55:49.694941 (+ 2375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:49.694974 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:49.695601 (+ 627us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:49.696128 (+ 527us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:49.696702 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:49.698756 (+ 2054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:49.700243 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:49.701363 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:49.701560 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:49.702739 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:49.702746 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:49.702761 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:55:49.703506 (+ 745us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:49.703604 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:55:49.703632 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:49.703751 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:49.704026 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 04:55:49.704363 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":463,"threads_started":1}
I20260925 04:55:49.713506 15655 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76350eb59ef944948e16327cd1a3081f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76350eb59ef944948e16327cd1a3081f" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 35249 } }
I20260925 04:55:49.713853 15655 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76350eb59ef944948e16327cd1a3081f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:55:49.713932 15655 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76350eb59ef944948e16327cd1a3081f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76350eb59ef944948e16327cd1a3081f, State: Initialized, Role: FOLLOWER
I20260925 04:55:49.714417 15655 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76350eb59ef944948e16327cd1a3081f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76350eb59ef944948e16327cd1a3081f" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 35249 } }
I20260925 04:55:49.714608 15655 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76350eb59ef944948e16327cd1a3081f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:55:49.714704 15655 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76350eb59ef944948e16327cd1a3081f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:55:49.714833 15655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76350eb59ef944948e16327cd1a3081f [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:55:49.717027 15655 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76350eb59ef944948e16327cd1a3081f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76350eb59ef944948e16327cd1a3081f" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 35249 } }
I20260925 04:55:49.717489 15655 leader_election.cc:302] T 00000000000000000000000000000000 P 76350eb59ef944948e16327cd1a3081f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76350eb59ef944948e16327cd1a3081f; no voters:
I20260925 04:55:49.718021 15655 leader_election.cc:288] T 00000000000000000000000000000000 P 76350eb59ef944948e16327cd1a3081f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:55:49.718214 15660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76350eb59ef944948e16327cd1a3081f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:55:49.719077 15660 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76350eb59ef944948e16327cd1a3081f [term 1 LEADER]: Becoming Leader. State: Replica: 76350eb59ef944948e16327cd1a3081f, State: Running, Role: LEADER
I20260925 04:55:49.719596 15660 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76350eb59ef944948e16327cd1a3081f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76350eb59ef944948e16327cd1a3081f" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 35249 } }
I20260925 04:55:49.720497 15655 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76350eb59ef944948e16327cd1a3081f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:55:49.722476 15661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76350eb59ef944948e16327cd1a3081f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76350eb59ef944948e16327cd1a3081f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76350eb59ef944948e16327cd1a3081f" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 35249 } } }
I20260925 04:55:49.722885 15661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76350eb59ef944948e16327cd1a3081f [sys.catalog]: This master's current role is: LEADER
I20260925 04:55:49.722815 15662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76350eb59ef944948e16327cd1a3081f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76350eb59ef944948e16327cd1a3081f. Latest consensus state: current_term: 1 leader_uuid: "76350eb59ef944948e16327cd1a3081f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76350eb59ef944948e16327cd1a3081f" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 35249 } } }
I20260925 04:55:49.723140 15662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76350eb59ef944948e16327cd1a3081f [sys.catalog]: This master's current role is: LEADER
I20260925 04:55:49.725185 15668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:55:49.730717 15668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:55:49.739921 15668 catalog_manager.cc:1347] Generated new cluster ID: 6d75648739ea412da29315c163a023a1
I20260925 04:55:49.740113 15668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:55:49.756886 15668 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:55:49.758692 15668 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:55:49.769488 15668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76350eb59ef944948e16327cd1a3081f: Generated new TSK 0
I20260925 04:55:49.770618 15668 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:55:49.777045 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.129:0
--local_ip_for_outbound_sockets=127.6.29.129
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:35249
--builtin_ntp_servers=127.6.29.148:39641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 04:55:50.111970 15679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:55:50.112269 15679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:55:50.112355 15679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:55:50.119808 15679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:55:50.119922 15679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:55:50.120030 15679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.129
I20260925 04:55:50.129817 15679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:39641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:35249
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.129
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:55:50.131134 15679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:55:50.132966 15679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:55:50.147061 15684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:50.147100 15685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:50.147136 15687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:55:50.148016 15679 server_base.cc:1034] running on GCE node
I20260925 04:55:50.148664 15679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:55:50.149685 15679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:55:50.150942 15679 hybrid_clock.cc:648] HybridClock initialized: now 1790312150150887 us; error 45 us; skew 500 ppm
I20260925 04:55:50.151274 15679 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 04:55:50.153602 15679 webserver.cc:460] Webserver started at http://127.6.29.129:41675/ using document root <none> and password file <none>
I20260925 04:55:50.154316 15679 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:50.154466 15679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:50.154796 15679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:55:50.157697 15679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/data/instance:
uuid: "f493600906534baaa1249e0e42a9c8cd"
format_stamp: "Formatted at 2026-09-25 04:55:50 on dist-test-slave-xzjr"
server_key: "88069e70114d8f52c27de60bc63a63a4"
server_key_iv: "84afdbfe25cb526ab4b88eee819e0876"
server_key_version: "encryptionkey@0"
I20260925 04:55:50.158501 15679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance:
uuid: "f493600906534baaa1249e0e42a9c8cd"
format_stamp: "Formatted at 2026-09-25 04:55:50 on dist-test-slave-xzjr"
server_key: "88069e70114d8f52c27de60bc63a63a4"
server_key_iv: "84afdbfe25cb526ab4b88eee819e0876"
server_key_version: "encryptionkey@0"
I20260925 04:55:50.163672 15679 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 04:55:50.167062 15693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:50.168490 15679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:55:50.168669 15679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/wal
uuid: "f493600906534baaa1249e0e42a9c8cd"
format_stamp: "Formatted at 2026-09-25 04:55:50 on dist-test-slave-xzjr"
server_key: "88069e70114d8f52c27de60bc63a63a4"
server_key_iv: "84afdbfe25cb526ab4b88eee819e0876"
server_key_version: "encryptionkey@0"
I20260925 04:55:50.168881 15679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:50.186353 15679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:55:50.187271 15679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:55:50.187593 15679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:55:50.188645 15679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:55:50.190176 15679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:55:50.190297 15679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:50.190431 15679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:55:50.190519 15679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:50.228592 15679 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.129:39467
I20260925 04:55:50.228693 15805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.129:39467 every 8 connection(s)
I20260925 04:55:50.229950 15679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
I20260925 04:55:50.231992 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 15679
I20260925 04:55:50.232201 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance
I20260925 04:55:50.232970 6262 external_mini_cluster.cc:1428] Setting key a22cb45a3b67a578e857cc21ec10498e
I20260925 04:55:50.235992 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.130:0
--local_ip_for_outbound_sockets=127.6.29.130
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:35249
--builtin_ntp_servers=127.6.29.148:39641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 04:55:50.243443 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:50.232691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:44595 (local address 127.6.29.190:35249)
0925 04:55:50.233156 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:50.233163 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:50.234555 (+ 1392us) server_negotiation.cc:407] Connection header received
0925 04:55:50.236710 (+ 2155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:50.236716 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:50.236797 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:50.236953 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:50.238165 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:50.238875 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:50.240541 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:50.240950 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:50.241092 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:50.242697 (+ 1605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:50.242702 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:50.242705 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:55:50.242723 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:50.242775 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:55:50.242788 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:50.242862 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:50.243109 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 04:55:50.243185 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":209,"threads_started":1}
I20260925 04:55:50.244997 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:50.233334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:35249 (local address 127.6.29.129:44595)
0925 04:55:50.234388 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:50.234439 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:50.236409 (+ 1970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:50.237303 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:50.237312 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:50.237405 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:50.237994 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:50.238013 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:50.239027 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:50.239033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:50.240394 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:50.240407 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:50.241181 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:50.241188 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:50.241485 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:50.242303 (+ 818us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:55:50.242309 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:50.242367 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:55:50.242481 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:50.242888 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:50.242900 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:50.242912 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 04:55:50.243097 (+ 185us) client_negotiation.cc:240] Negotiation successful
0925 04:55:50.243529 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":323,"threads_started":1}
I20260925 04:55:50.246516 15806 heartbeater.cc:347] Connected to a master server at 127.6.29.190:35249
I20260925 04:55:50.246822 15806 heartbeater.cc:464] Registering TS with master...
I20260925 04:55:50.247587 15806 heartbeater.cc:511] Master 127.6.29.190:35249 requested a full tablet report, sending...
I20260925 04:55:50.249696 15620 ts_manager.cc:194] Registered new tserver with Master: f493600906534baaa1249e0e42a9c8cd (127.6.29.129:39467)
I20260925 04:55:50.251173 15620 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.29.129:44595
W20260925 04:55:50.565820 15810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:55:50.566123 15810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:55:50.566257 15810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:55:50.573690 15810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:55:50.573807 15810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:55:50.573974 15810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.130
I20260925 04:55:50.583320 15810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:39641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:35249
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.130
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:55:50.584705 15810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:55:50.586520 15810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:55:50.600020 15815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:50.600054 15818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:50.599992 15816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:55:50.601028 15810 server_base.cc:1034] running on GCE node
I20260925 04:55:50.601588 15810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:55:50.602492 15810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:55:50.603691 15810 hybrid_clock.cc:648] HybridClock initialized: now 1790312150603637 us; error 44 us; skew 500 ppm
I20260925 04:55:50.604034 15810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 04:55:50.606393 15810 webserver.cc:460] Webserver started at http://127.6.29.130:34667/ using document root <none> and password file <none>
I20260925 04:55:50.607070 15810 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:50.607164 15810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:50.607429 15810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:55:50.610407 15810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/data/instance:
uuid: "1370d33db2594f8e8d0d2ea59b0d26da"
format_stamp: "Formatted at 2026-09-25 04:55:50 on dist-test-slave-xzjr"
server_key: "9864644224335fa243fd623886bd7fbc"
server_key_iv: "cf1ea815e5d5786725bb89315f10c057"
server_key_version: "encryptionkey@0"
I20260925 04:55:50.611188 15810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance:
uuid: "1370d33db2594f8e8d0d2ea59b0d26da"
format_stamp: "Formatted at 2026-09-25 04:55:50 on dist-test-slave-xzjr"
server_key: "9864644224335fa243fd623886bd7fbc"
server_key_iv: "cf1ea815e5d5786725bb89315f10c057"
server_key_version: "encryptionkey@0"
I20260925 04:55:50.616370 15810 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260925 04:55:50.619951 15824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:50.621551 15810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:55:50.621738 15810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/wal
uuid: "1370d33db2594f8e8d0d2ea59b0d26da"
format_stamp: "Formatted at 2026-09-25 04:55:50 on dist-test-slave-xzjr"
server_key: "9864644224335fa243fd623886bd7fbc"
server_key_iv: "cf1ea815e5d5786725bb89315f10c057"
server_key_version: "encryptionkey@0"
I20260925 04:55:50.621959 15810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:50.635509 15810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:55:50.636454 15810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:55:50.636756 15810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:55:50.637687 15810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:55:50.639093 15810 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:55:50.639212 15810 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:50.639339 15810 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:55:50.639429 15810 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:50.673368 15810 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.130:36075
I20260925 04:55:50.673415 15936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.130:36075 every 8 connection(s)
I20260925 04:55:50.674774 15810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
I20260925 04:55:50.679751 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 15810
I20260925 04:55:50.680009 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance
I20260925 04:55:50.680425 6262 external_mini_cluster.cc:1428] Setting key b24e4e680e19758869d74812ac975596
I20260925 04:55:50.683132 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.131:0
--local_ip_for_outbound_sockets=127.6.29.131
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:35249
--builtin_ntp_servers=127.6.29.148:39641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 04:55:50.690343 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:50.677314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:58893 (local address 127.6.29.190:35249)
0925 04:55:50.677522 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:50.677528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:50.679143 (+ 1615us) server_negotiation.cc:407] Connection header received
0925 04:55:50.681577 (+ 2434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:50.681582 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:50.681654 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:50.681761 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:50.683180 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:50.684353 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:50.686872 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:50.687540 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:50.687660 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:50.689474 (+ 1814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:50.689481 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:50.689486 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:55:50.689508 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:50.689545 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:55:50.689560 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:50.689649 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:50.689914 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 04:55:50.690014 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 04:55:50.691864 15939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:50.677860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:35249 (local address 127.6.29.130:58893)
0925 04:55:50.678950 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:50.679003 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:50.681275 (+ 2272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:50.682004 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:50.682016 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:50.682153 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:50.682961 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:50.682988 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:50.684525 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:50.684533 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:50.686691 (+ 2158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:50.686709 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:50.687825 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:50.687834 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:50.688199 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:50.689098 (+ 899us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:55:50.689104 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:50.689163 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:55:50.689269 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:50.689668 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:50.689681 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:50.689692 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 04:55:50.689890 (+ 198us) client_negotiation.cc:240] Negotiation successful
0925 04:55:50.690362 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":304,"threads_started":1}
I20260925 04:55:50.693370 15937 heartbeater.cc:347] Connected to a master server at 127.6.29.190:35249
I20260925 04:55:50.693701 15937 heartbeater.cc:464] Registering TS with master...
I20260925 04:55:50.694466 15937 heartbeater.cc:511] Master 127.6.29.190:35249 requested a full tablet report, sending...
I20260925 04:55:50.695904 15620 ts_manager.cc:194] Registered new tserver with Master: 1370d33db2594f8e8d0d2ea59b0d26da (127.6.29.130:36075)
I20260925 04:55:50.696715 15620 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.29.130:58893
W20260925 04:55:51.018008 15940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:55:51.018307 15940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:55:51.018512 15940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:55:51.025990 15940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:55:51.026155 15940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:55:51.026352 15940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.131
I20260925 04:55:51.036036 15940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:39641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:35249
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.131
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:55:51.037408 15940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:55:51.039222 15940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:55:51.052899 15945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:51.052943 15946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:51.052987 15948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:55:51.053879 15940 server_base.cc:1034] running on GCE node
I20260925 04:55:51.054512 15940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:55:51.055511 15940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:55:51.056710 15940 hybrid_clock.cc:648] HybridClock initialized: now 1790312151056630 us; error 67 us; skew 500 ppm
I20260925 04:55:51.057045 15940 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 04:55:51.059437 15940 webserver.cc:460] Webserver started at http://127.6.29.131:40375/ using document root <none> and password file <none>
I20260925 04:55:51.060240 15940 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:51.060353 15940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:51.060688 15940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:55:51.063575 15940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/data/instance:
uuid: "cd4d849882f64866adf19b24a95157c7"
format_stamp: "Formatted at 2026-09-25 04:55:51 on dist-test-slave-xzjr"
server_key: "a2bfa41c7a9a1dc3917d62a2c4475f4b"
server_key_iv: "87ae28cb8a7f44356fa8281bce214734"
server_key_version: "encryptionkey@0"
I20260925 04:55:51.064452 15940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance:
uuid: "cd4d849882f64866adf19b24a95157c7"
format_stamp: "Formatted at 2026-09-25 04:55:51 on dist-test-slave-xzjr"
server_key: "a2bfa41c7a9a1dc3917d62a2c4475f4b"
server_key_iv: "87ae28cb8a7f44356fa8281bce214734"
server_key_version: "encryptionkey@0"
I20260925 04:55:51.069638 15940 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 04:55:51.073104 15954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:51.074518 15940 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:55:51.074690 15940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/wal
uuid: "cd4d849882f64866adf19b24a95157c7"
format_stamp: "Formatted at 2026-09-25 04:55:51 on dist-test-slave-xzjr"
server_key: "a2bfa41c7a9a1dc3917d62a2c4475f4b"
server_key_iv: "87ae28cb8a7f44356fa8281bce214734"
server_key_version: "encryptionkey@0"
I20260925 04:55:51.074906 15940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:51.096272 15940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:55:51.097139 15940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:55:51.097447 15940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:55:51.098457 15940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:55:51.100024 15940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:55:51.100109 15940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:51.100262 15940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:55:51.100354 15940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:51.134402 15940 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.131:40529
I20260925 04:55:51.134493 16066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.131:40529 every 8 connection(s)
I20260925 04:55:51.135696 15940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
I20260925 04:55:51.140770 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 15940
I20260925 04:55:51.140974 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance
I20260925 04:55:51.141397 6262 external_mini_cluster.cc:1428] Setting key 88958e3650b037e9bb574888ee6d7561
I20260925 04:55:51.148437 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:51.138346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:40987 (local address 127.6.29.190:35249)
0925 04:55:51.138536 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:51.138540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:51.140057 (+ 1517us) server_negotiation.cc:407] Connection header received
0925 04:55:51.141906 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:51.141911 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:51.141978 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:51.142063 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:51.143288 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:51.144204 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:51.145754 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:51.146162 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:51.146270 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:51.147659 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:51.147664 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:51.147667 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:55:51.147686 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:51.147718 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:55:51.147730 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:51.147900 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:51.148114 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 04:55:51.148191 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 04:55:51.149439 16069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:51.138898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:35249 (local address 127.6.29.131:40987)
0925 04:55:51.139905 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:51.139942 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:51.141634 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:51.142336 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:51.142361 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:51.142491 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:51.143140 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:51.143158 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:51.144356 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:51.144362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:51.145594 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:51.145606 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:51.146358 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:51.146363 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:51.146566 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:51.147328 (+ 762us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:55:51.147330 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:51.147369 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:55:51.147452 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:51.147902 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:51.147918 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:51.147925 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 04:55:51.148080 (+ 155us) client_negotiation.cc:240] Negotiation successful
0925 04:55:51.148389 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":300,"threads_started":1}
I20260925 04:55:51.150806 16067 heartbeater.cc:347] Connected to a master server at 127.6.29.190:35249
I20260925 04:55:51.151110 16067 heartbeater.cc:464] Registering TS with master...
I20260925 04:55:51.152046 16067 heartbeater.cc:511] Master 127.6.29.190:35249 requested a full tablet report, sending...
I20260925 04:55:51.153456 15620 ts_manager.cc:194] Registered new tserver with Master: cd4d849882f64866adf19b24a95157c7 (127.6.29.131:40529)
I20260925 04:55:51.154197 15620 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.29.131:40987
I20260925 04:55:51.156802 6262 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 04:55:51.162616 15808 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:51.161236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37710 (local address 127.6.29.190:35249)
0925 04:55:51.161430 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:51.161434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:51.161825 (+ 391us) server_negotiation.cc:407] Connection header received
0925 04:55:51.161956 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:51.161960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:51.162030 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:51.162111 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:51.162439 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37710: BlockingRecv error: recv got EOF from 127.0.0.1:37710 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 04:55:51.163097 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 15679
I20260925 04:55:51.171644 16076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:51.164159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47614 (local address 127.6.29.129:39467)
0925 04:55:51.164738 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:51.164744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:51.164769 (+ 25us) server_negotiation.cc:407] Connection header received
0925 04:55:51.164828 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:51.164834 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:51.165039 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:51.165240 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:51.165627 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:51.166622 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:51.167932 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:51.168488 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:51.168627 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:51.169778 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:51.169790 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:51.169808 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:55:51.170741 (+ 933us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:51.170807 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:55:51.170834 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:51.170951 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:51.171110 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 04:55:51.171274 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":287,"threads_started":1}
I20260925 04:55:51.172905 15801 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:55:51.254211 15806 heartbeater.cc:503] Master 127.6.29.190:35249 was elected leader, sending a full tablet report...
I20260925 04:55:51.255574 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 15679
W20260925 04:55:51.271608 15606 connection.cc:535] server connection from 127.6.29.129:44595 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 04:55:51.271942 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 15810
I20260925 04:55:51.279578 16078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:51.272838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60506 (local address 127.6.29.130:36075)
0925 04:55:51.273320 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:51.273327 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:51.273354 (+ 27us) server_negotiation.cc:407] Connection header received
0925 04:55:51.273419 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:51.273425 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:51.273669 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:51.273803 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:51.274170 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:51.275145 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:51.276486 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:51.277042 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:51.277144 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:51.278090 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:51.278098 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:51.278112 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:55:51.278736 (+ 624us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:51.278783 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:55:51.278808 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:51.278898 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:51.279068 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 04:55:51.279246 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":227,"threads_started":1}
I20260925 04:55:51.280845 15932 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:55:51.364884 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 15810
I20260925 04:55:51.381870 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 15940
I20260925 04:55:51.389989 16080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:51.382892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35704 (local address 127.6.29.131:40529)
0925 04:55:51.383495 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:51.383503 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:51.383533 (+ 30us) server_negotiation.cc:407] Connection header received
0925 04:55:51.383627 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:51.383633 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:51.383929 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:51.384056 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:51.384460 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:51.385504 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:51.386775 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:51.387361 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:51.387453 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:51.388556 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:51.388587 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:51.388602 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:55:51.389174 (+ 572us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:51.389219 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:55:51.389237 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:51.389323 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:51.389474 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 04:55:51.389626 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":319,"threads_started":1}
I20260925 04:55:51.391286 16062 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:55:51.473490 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 15940
I20260925 04:55:51.489673 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 15589
I20260925 04:55:51.491086 15650 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:55:51.592609 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 15589
2026-09-25T04:55:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2428 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T04:55:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:55:51Z Disabled control of system clock
I20260925 04:55:51.665812 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:33503
--webserver_interface=127.6.29.190
--webserver_port=0
--builtin_ntp_servers=127.6.29.148:36693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:33503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 04:55:51.994776 16086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:55:51.995085 16086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:55:51.995235 16086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:55:52.002570 16086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:55:52.002679 16086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:55:52.002755 16086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:55:52.002843 16086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:55:52.002933 16086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:55:52.012457 16086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:36693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:33503
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:33503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:55:52.013835 16086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:55:52.015677 16086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:55:52.028297 16094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:52.028331 16091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:52.028321 16092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:55:52.029251 16086 server_base.cc:1034] running on GCE node
I20260925 04:55:52.030097 16086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:55:52.031693 16086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:55:52.032974 16086 hybrid_clock.cc:648] HybridClock initialized: now 1790312152032932 us; error 36 us; skew 500 ppm
I20260925 04:55:52.033365 16086 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 04:55:52.035872 16086 webserver.cc:460] Webserver started at http://127.6.29.190:45599/ using document root <none> and password file <none>
I20260925 04:55:52.036612 16086 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:52.036736 16086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:52.037066 16086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:55:52.041482 16086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "d1df58e5799040408f1feb438bd09879"
format_stamp: "Formatted at 2026-09-25 04:55:52 on dist-test-slave-xzjr"
server_key: "082407144b279d4d0b666c0872280aa1"
server_key_iv: "0861fe18b63782ee68166549f8203412"
server_key_version: "encryptionkey@0"
I20260925 04:55:52.042308 16086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "d1df58e5799040408f1feb438bd09879"
format_stamp: "Formatted at 2026-09-25 04:55:52 on dist-test-slave-xzjr"
server_key: "082407144b279d4d0b666c0872280aa1"
server_key_iv: "0861fe18b63782ee68166549f8203412"
server_key_version: "encryptionkey@0"
I20260925 04:55:52.047632 16086 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 04:55:52.051159 16100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:52.052577 16086 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:55:52.052784 16086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "d1df58e5799040408f1feb438bd09879"
format_stamp: "Formatted at 2026-09-25 04:55:52 on dist-test-slave-xzjr"
server_key: "082407144b279d4d0b666c0872280aa1"
server_key_iv: "0861fe18b63782ee68166549f8203412"
server_key_version: "encryptionkey@0"
I20260925 04:55:52.053020 16086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:52.059015 16086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:55:52.059904 16086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:55:52.060205 16086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:55:52.078188 16086 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:33503
I20260925 04:55:52.078220 16151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:33503 every 8 connection(s)
I20260925 04:55:52.079555 16086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:55:52.083314 16152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:55:52.087900 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 16086
I20260925 04:55:52.088115 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 04:55:52.088889 6262 external_mini_cluster.cc:1428] Setting key 220e2d3e610db767214c46225802208b
I20260925 04:55:52.095305 16152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1df58e5799040408f1feb438bd09879: Bootstrap starting.
I20260925 04:55:52.097523 16155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:52.091124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60536 (local address 127.6.29.190:33503)
0925 04:55:52.092227 (+ 1103us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:52.092238 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:52.092281 (+ 43us) server_negotiation.cc:407] Connection header received
0925 04:55:52.093953 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:52.093976 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:52.094394 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:52.094738 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:52.095905 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:52.095913 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:52.095926 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:55:52.095940 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:52.096000 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:55:52.096016 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:52.096182 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:52.096308 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 04:55:52.096597 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":391,"threads_started":1}
I20260925 04:55:52.098997 16152 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1df58e5799040408f1feb438bd09879: Neither blocks nor log segments found. Creating new log.
I20260925 04:55:52.100095 16152 log.cc:824] T 00000000000000000000000000000000 P d1df58e5799040408f1feb438bd09879: Log is configured to *not* fsync() on all Append() calls
I20260925 04:55:52.102797 16152 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1df58e5799040408f1feb438bd09879: No bootstrap required, opened a new log
I20260925 04:55:52.110045 16152 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1df58e5799040408f1feb438bd09879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1df58e5799040408f1feb438bd09879" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33503 } }
I20260925 04:55:52.110497 16152 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1df58e5799040408f1feb438bd09879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:55:52.110620 16152 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1df58e5799040408f1feb438bd09879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1df58e5799040408f1feb438bd09879, State: Initialized, Role: FOLLOWER
I20260925 04:55:52.111317 16152 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1df58e5799040408f1feb438bd09879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1df58e5799040408f1feb438bd09879" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33503 } }
I20260925 04:55:52.111545 16152 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1df58e5799040408f1feb438bd09879 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:55:52.111680 16152 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1df58e5799040408f1feb438bd09879 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:55:52.111891 16152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1df58e5799040408f1feb438bd09879 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:55:52.113978 16152 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1df58e5799040408f1feb438bd09879 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1df58e5799040408f1feb438bd09879" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33503 } }
I20260925 04:55:52.114439 16152 leader_election.cc:302] T 00000000000000000000000000000000 P d1df58e5799040408f1feb438bd09879 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1df58e5799040408f1feb438bd09879; no voters:
I20260925 04:55:52.114979 16152 leader_election.cc:288] T 00000000000000000000000000000000 P d1df58e5799040408f1feb438bd09879 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:55:52.115217 16157 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1df58e5799040408f1feb438bd09879 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:55:52.115981 16157 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1df58e5799040408f1feb438bd09879 [term 1 LEADER]: Becoming Leader. State: Replica: d1df58e5799040408f1feb438bd09879, State: Running, Role: LEADER
I20260925 04:55:52.116494 16157 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1df58e5799040408f1feb438bd09879 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1df58e5799040408f1feb438bd09879" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33503 } }
I20260925 04:55:52.117105 16152 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1df58e5799040408f1feb438bd09879 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:55:52.119191 16158 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1df58e5799040408f1feb438bd09879 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1df58e5799040408f1feb438bd09879" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1df58e5799040408f1feb438bd09879" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33503 } } }
I20260925 04:55:52.119541 16158 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1df58e5799040408f1feb438bd09879 [sys.catalog]: This master's current role is: LEADER
I20260925 04:55:52.119915 16159 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1df58e5799040408f1feb438bd09879 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1df58e5799040408f1feb438bd09879. Latest consensus state: current_term: 1 leader_uuid: "d1df58e5799040408f1feb438bd09879" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1df58e5799040408f1feb438bd09879" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33503 } } }
I20260925 04:55:52.120319 16159 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1df58e5799040408f1feb438bd09879 [sys.catalog]: This master's current role is: LEADER
I20260925 04:55:52.124316 16163 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:55:52.129954 16163 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:55:52.138942 16163 catalog_manager.cc:1347] Generated new cluster ID: b5d7e1bc07d044c4856808441cc08ef2
I20260925 04:55:52.139060 16163 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:55:52.172101 16163 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:55:52.172619 16163 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:55:52.183002 16163 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1df58e5799040408f1feb438bd09879: Generated new TSK 0
I20260925 04:55:52.184016 16163 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:55:52.194509 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.129:0
--local_ip_for_outbound_sockets=127.6.29.129
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:33503
--builtin_ntp_servers=127.6.29.148:36693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 04:55:52.546151 16177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:55:52.546458 16177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:55:52.546552 16177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:55:52.553970 16177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:55:52.554116 16177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:55:52.554229 16177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.129
I20260925 04:55:52.563941 16177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:36693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:33503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.129
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:55:52.565265 16177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:55:52.566988 16177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:55:52.580616 16185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:52.580662 16183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:52.580595 16182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:55:52.581575 16177 server_base.cc:1034] running on GCE node
I20260925 04:55:52.582228 16177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:55:52.583284 16177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:55:52.584547 16177 hybrid_clock.cc:648] HybridClock initialized: now 1790312152584487 us; error 51 us; skew 500 ppm
I20260925 04:55:52.584939 16177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 04:55:52.587381 16177 webserver.cc:460] Webserver started at http://127.6.29.129:42597/ using document root <none> and password file <none>
I20260925 04:55:52.588200 16177 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:52.588336 16177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:52.588666 16177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:55:52.591527 16177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/data/instance:
uuid: "600b1bf7ebeb442fae3514b5938eb9a4"
format_stamp: "Formatted at 2026-09-25 04:55:52 on dist-test-slave-xzjr"
server_key: "dc0bc356a9028ab39dfd7bbf2b968015"
server_key_iv: "83af27f8ae7057f76e95a715743838e9"
server_key_version: "encryptionkey@0"
I20260925 04:55:52.592406 16177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance:
uuid: "600b1bf7ebeb442fae3514b5938eb9a4"
format_stamp: "Formatted at 2026-09-25 04:55:52 on dist-test-slave-xzjr"
server_key: "dc0bc356a9028ab39dfd7bbf2b968015"
server_key_iv: "83af27f8ae7057f76e95a715743838e9"
server_key_version: "encryptionkey@0"
I20260925 04:55:52.597874 16177 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 04:55:52.601666 16191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:52.603145 16177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:55:52.603286 16177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/wal
uuid: "600b1bf7ebeb442fae3514b5938eb9a4"
format_stamp: "Formatted at 2026-09-25 04:55:52 on dist-test-slave-xzjr"
server_key: "dc0bc356a9028ab39dfd7bbf2b968015"
server_key_iv: "83af27f8ae7057f76e95a715743838e9"
server_key_version: "encryptionkey@0"
I20260925 04:55:52.603529 16177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:52.609493 16177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:55:52.610322 16177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:55:52.610602 16177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:55:52.611725 16177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:55:52.613225 16177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:55:52.613339 16177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:52.613477 16177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:55:52.613579 16177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:52.646438 16177 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.129:45053
I20260925 04:55:52.646472 16303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.129:45053 every 8 connection(s)
I20260925 04:55:52.647894 16177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
I20260925 04:55:52.652487 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 16177
I20260925 04:55:52.652678 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance
I20260925 04:55:52.653489 6262 external_mini_cluster.cc:1428] Setting key f621e97c8328a099b7d7519501bcaa3f
I20260925 04:55:52.656783 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.130:0
--local_ip_for_outbound_sockets=127.6.29.130
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:33503
--builtin_ntp_servers=127.6.29.148:36693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 04:55:52.657243 16306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:52.650574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:42501 (local address 127.6.29.190:33503)
0925 04:55:52.651037 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:52.651042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:52.652297 (+ 1255us) server_negotiation.cc:407] Connection header received
0925 04:55:52.654735 (+ 2438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:52.654740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:52.654803 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:52.654937 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:52.656433 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:52.656437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:52.656441 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:55:52.656456 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:52.656526 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:55:52.656548 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:52.656676 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:52.656900 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 04:55:52.656975 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":253,"threads_started":1}
I20260925 04:55:52.658691 16307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:52.651052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:33503 (local address 127.6.29.129:42501)
0925 04:55:52.652093 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:52.652146 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:52.654388 (+ 2242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:52.655131 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:52.655138 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:52.655233 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:52.656084 (+ 851us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:55:52.656088 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:52.656138 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:55:52.656253 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:52.656649 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:52.656659 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:52.656667 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 04:55:52.656965 (+ 298us) client_negotiation.cc:240] Negotiation successful
0925 04:55:52.657313 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":261,"threads_started":1}
I20260925 04:55:52.660279 16304 heartbeater.cc:347] Connected to a master server at 127.6.29.190:33503
I20260925 04:55:52.660617 16304 heartbeater.cc:464] Registering TS with master...
I20260925 04:55:52.661335 16304 heartbeater.cc:511] Master 127.6.29.190:33503 requested a full tablet report, sending...
I20260925 04:55:52.663560 16117 ts_manager.cc:194] Registered new tserver with Master: 600b1bf7ebeb442fae3514b5938eb9a4 (127.6.29.129:45053)
W20260925 04:55:52.988806 16308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:55:52.989133 16308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:55:52.989318 16308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:55:52.997243 16308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:55:52.997357 16308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:55:52.997565 16308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.130
I20260925 04:55:53.006975 16308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:36693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:33503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.130
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:55:53.008459 16308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:55:53.010257 16308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:55:53.024109 16313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:53.024084 16314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:53.024523 16316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:55:53.025727 16308 server_base.cc:1034] running on GCE node
I20260925 04:55:53.026439 16308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:55:53.027558 16308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:55:53.029906 16308 hybrid_clock.cc:648] HybridClock initialized: now 1790312153029827 us; error 108 us; skew 500 ppm
I20260925 04:55:53.030280 16308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 04:55:53.032994 16308 webserver.cc:460] Webserver started at http://127.6.29.130:41691/ using document root <none> and password file <none>
I20260925 04:55:53.033823 16308 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:53.033959 16308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:53.034307 16308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:55:53.037266 16308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/data/instance:
uuid: "9878910e0bcf477ebe34d61738e1ebad"
format_stamp: "Formatted at 2026-09-25 04:55:53 on dist-test-slave-xzjr"
server_key: "d15d5727c67cda268e88217c3218ba58"
server_key_iv: "ce65a8e2ce3309c86c460da4cfb3ec5b"
server_key_version: "encryptionkey@0"
I20260925 04:55:53.038092 16308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance:
uuid: "9878910e0bcf477ebe34d61738e1ebad"
format_stamp: "Formatted at 2026-09-25 04:55:53 on dist-test-slave-xzjr"
server_key: "d15d5727c67cda268e88217c3218ba58"
server_key_iv: "ce65a8e2ce3309c86c460da4cfb3ec5b"
server_key_version: "encryptionkey@0"
I20260925 04:55:53.043465 16308 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 04:55:53.047194 16322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:53.048693 16308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:55:53.048887 16308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/wal
uuid: "9878910e0bcf477ebe34d61738e1ebad"
format_stamp: "Formatted at 2026-09-25 04:55:53 on dist-test-slave-xzjr"
server_key: "d15d5727c67cda268e88217c3218ba58"
server_key_iv: "ce65a8e2ce3309c86c460da4cfb3ec5b"
server_key_version: "encryptionkey@0"
I20260925 04:55:53.049129 16308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:53.054860 16308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:55:53.055670 16308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:55:53.055979 16308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:55:53.056995 16308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:55:53.058473 16308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:55:53.058596 16308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:53.058734 16308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:55:53.058823 16308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:53.093500 16308 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.130:44927
I20260925 04:55:53.093518 16434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.130:44927 every 8 connection(s)
I20260925 04:55:53.094894 16308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
I20260925 04:55:53.101025 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 16308
I20260925 04:55:53.101203 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance
I20260925 04:55:53.101683 6262 external_mini_cluster.cc:1428] Setting key fb777d0dec56f00ca4a20b5618329072
I20260925 04:55:53.103533 16306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:53.097760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:53409 (local address 127.6.29.190:33503)
0925 04:55:53.097934 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:53.097938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:53.099340 (+ 1402us) server_negotiation.cc:407] Connection header received
0925 04:55:53.101093 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:53.101101 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:53.101175 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:53.101300 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:53.102796 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:53.102801 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:53.102806 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:55:53.102831 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:53.102859 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:55:53.102870 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:53.102992 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:53.103199 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 04:55:53.103265 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 04:55:53.104527 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.131:0
--local_ip_for_outbound_sockets=127.6.29.131
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:33503
--builtin_ntp_servers=127.6.29.148:36693
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 04:55:53.104784 16437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:53.098290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:33503 (local address 127.6.29.130:53409)
0925 04:55:53.099202 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:53.099244 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:53.100813 (+ 1569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:53.101526 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:53.101535 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:53.101661 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:53.102454 (+ 793us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:55:53.102460 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:53.102516 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:55:53.102632 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:53.102967 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:53.102977 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:53.102984 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 04:55:53.103271 (+ 287us) client_negotiation.cc:240] Negotiation successful
0925 04:55:53.103537 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":279,"threads_started":1}
I20260925 04:55:53.106205 16435 heartbeater.cc:347] Connected to a master server at 127.6.29.190:33503
I20260925 04:55:53.106511 16435 heartbeater.cc:464] Registering TS with master...
I20260925 04:55:53.107204 16435 heartbeater.cc:511] Master 127.6.29.190:33503 requested a full tablet report, sending...
I20260925 04:55:53.108805 16117 ts_manager.cc:194] Registered new tserver with Master: 9878910e0bcf477ebe34d61738e1ebad (127.6.29.130:44927)
W20260925 04:55:53.444568 16438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:55:53.444857 16438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:55:53.444958 16438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:55:53.452360 16438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:55:53.452472 16438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:55:53.452591 16438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.131
I20260925 04:55:53.462137 16438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:36693
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:33503
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.131
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:55:53.463423 16438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:55:53.465248 16438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:55:53.478322 16444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:53.478252 16443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:53.478976 16446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:55:53.479385 16438 server_base.cc:1034] running on GCE node
I20260925 04:55:53.480067 16438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:55:53.481173 16438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:55:53.482379 16438 hybrid_clock.cc:648] HybridClock initialized: now 1790312153482340 us; error 36 us; skew 500 ppm
I20260925 04:55:53.482719 16438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 04:55:53.485174 16438 webserver.cc:460] Webserver started at http://127.6.29.131:39545/ using document root <none> and password file <none>
I20260925 04:55:53.485906 16438 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:53.486023 16438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:53.486369 16438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:55:53.489394 16438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/data/instance:
uuid: "b00074d5cf10458a84e5b8f9b09eb508"
format_stamp: "Formatted at 2026-09-25 04:55:53 on dist-test-slave-xzjr"
server_key: "b19b986952708dbdbf4f82d41ea3104f"
server_key_iv: "3d9e29be582afe03c2f0dab34f46d1aa"
server_key_version: "encryptionkey@0"
I20260925 04:55:53.490212 16438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance:
uuid: "b00074d5cf10458a84e5b8f9b09eb508"
format_stamp: "Formatted at 2026-09-25 04:55:53 on dist-test-slave-xzjr"
server_key: "b19b986952708dbdbf4f82d41ea3104f"
server_key_iv: "3d9e29be582afe03c2f0dab34f46d1aa"
server_key_version: "encryptionkey@0"
I20260925 04:55:53.495447 16438 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 04:55:53.499096 16452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:53.500663 16438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:55:53.500881 16438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/wal
uuid: "b00074d5cf10458a84e5b8f9b09eb508"
format_stamp: "Formatted at 2026-09-25 04:55:53 on dist-test-slave-xzjr"
server_key: "b19b986952708dbdbf4f82d41ea3104f"
server_key_iv: "3d9e29be582afe03c2f0dab34f46d1aa"
server_key_version: "encryptionkey@0"
I20260925 04:55:53.501150 16438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:53.508445 16438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:55:53.509348 16438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:55:53.509631 16438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:55:53.510737 16438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:55:53.512503 16438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:55:53.512627 16438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:53.512742 16438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:55:53.512845 16438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:53.546352 16438 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.131:34493
I20260925 04:55:53.546452 16564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.131:34493 every 8 connection(s)
I20260925 04:55:53.547760 16438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
I20260925 04:55:53.548138 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 16438
I20260925 04:55:53.548326 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance
I20260925 04:55:53.549039 6262 external_mini_cluster.cc:1428] Setting key 9bb1b243785aa7979565a8fe34893a65
I20260925 04:55:53.556439 16306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:53.550828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:57821 (local address 127.6.29.190:33503)
0925 04:55:53.550989 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:53.550997 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:53.552645 (+ 1648us) server_negotiation.cc:407] Connection header received
0925 04:55:53.554314 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:53.554318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:53.554368 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:53.554452 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:53.555695 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:53.555700 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:53.555703 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:55:53.555717 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:53.555746 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:55:53.555802 (+ 56us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:53.555903 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:53.556102 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 04:55:53.556204 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 04:55:53.557348 16567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:53.551406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:33503 (local address 127.6.29.131:57821)
0925 04:55:53.552498 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:53.552541 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:53.554085 (+ 1544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:53.554659 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:53.554667 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:53.554756 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:53.555385 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:55:53.555387 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:53.555424 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:55:53.555516 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:53.555907 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:53.555921 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:53.555928 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 04:55:53.556160 (+ 232us) client_negotiation.cc:240] Negotiation successful
0925 04:55:53.556351 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":376,"threads_started":1}
I20260925 04:55:53.558802 16565 heartbeater.cc:347] Connected to a master server at 127.6.29.190:33503
I20260925 04:55:53.559108 16565 heartbeater.cc:464] Registering TS with master...
I20260925 04:55:53.559695 16565 heartbeater.cc:511] Master 127.6.29.190:33503 requested a full tablet report, sending...
I20260925 04:55:53.561088 16117 ts_manager.cc:194] Registered new tserver with Master: b00074d5cf10458a84e5b8f9b09eb508 (127.6.29.131:34493)
I20260925 04:55:53.564497 6262 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 04:55:53.570549 16306 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 04:55:53.568990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60548 (local address 127.6.29.190:33503)
0925 04:55:53.569163 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:53.569168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:53.569690 (+ 522us) server_negotiation.cc:407] Connection header received
0925 04:55:53.569810 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:53.569813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:53.569861 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:53.569941 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:53.570261 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60548: BlockingRecv error: recv got EOF from 127.0.0.1:60548 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 04:55:53.571724 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 16177
I20260925 04:55:53.575412 16574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:53.572778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44142 (local address 127.6.29.129:45053)
0925 04:55:53.573326 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:53.573339 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:53.573365 (+ 26us) server_negotiation.cc:407] Connection header received
0925 04:55:53.573421 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:53.573427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:53.573634 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:53.573753 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:53.574619 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:53.574626 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:53.574638 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:55:53.574656 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:53.574698 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:55:53.574712 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:53.574832 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:53.574963 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 04:55:53.575103 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":282,"threads_started":1}
I20260925 04:55:53.576843 16299 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:55:53.660071 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 16177
I20260925 04:55:53.676394 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 16308
I20260925 04:55:53.688302 16576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:53.677417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56554 (local address 127.6.29.130:44927)
0925 04:55:53.677952 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:53.677959 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:53.677986 (+ 27us) server_negotiation.cc:407] Connection header received
0925 04:55:53.678050 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:53.678056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:53.678268 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:53.678407 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:53.687350 (+ 8943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:53.687359 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:53.687374 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:55:53.687393 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:53.687469 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:55:53.687485 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:53.687612 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:53.687833 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 04:55:53.687987 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":258,"threads_started":1}
I20260925 04:55:53.689692 16430 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:55:53.771613 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 16308
I20260925 04:55:53.788604 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 16438
I20260925 04:55:53.792739 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:53.789712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36602 (local address 127.6.29.131:34493)
0925 04:55:53.790374 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:53.790388 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:53.790419 (+ 31us) server_negotiation.cc:407] Connection header received
0925 04:55:53.790488 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:53.790494 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:53.790715 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:53.790842 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:53.791992 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:53.791998 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:53.792010 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:55:53.792027 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:53.792065 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:55:53.792079 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:53.792208 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:53.792273 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 04:55:53.792382 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":350,"threads_started":1}
I20260925 04:55:53.794003 16560 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:55:53.875577 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 16438
I20260925 04:55:53.891700 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 16086
I20260925 04:55:53.892910 16147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:55:53.994908 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 16086
2026-09-25T04:55:54Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2402 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:55:54 dist-test-slave-xzjr krb5kdc[16583](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:55:54 dist-test-slave-xzjr krb5kdc[16583](info): set up 2 sockets
Sep 25 04:55:54 dist-test-slave-xzjr krb5kdc[16583](info): commencing operation
krb5kdc: starting...
W20260925 04:55:55.109404 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 04:55:55 dist-test-slave-xzjr krb5kdc[16583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:55:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:55:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.29.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
I20260925 04:55:55.352180 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:33109
--webserver_interface=127.6.29.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:34737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:33109
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:55:55.685334 16596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:55:55.685658 16596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:55:55.685833 16596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:55:55.693182 16596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:55:55.693302 16596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:55:55.693395 16596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:55:55.693500 16596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:55:55.693598 16596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:55:55.703893 16596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:34737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:33109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:33109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:55:55.705329 16596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:55:55.706686 16596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:55:55.718721 16604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:55.718743 16601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:55.718724 16602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:55:55.719606 16596 server_base.cc:1034] running on GCE node
I20260925 04:55:55.720443 16596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:55:55.721880 16596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:55:55.723083 16596 hybrid_clock.cc:648] HybridClock initialized: now 1790312155723022 us; error 55 us; skew 500 ppm
I20260925 04:55:55.723465 16596 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:55:55 dist-test-slave-xzjr krb5kdc[16583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312155, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:55:55.726608 16596 init.cc:377] Logged in from keytab as kudu/127.6.29.190@KRBTEST.COM (short username kudu)
I20260925 04:55:55.728084 16596 webserver.cc:460] Webserver started at http://127.6.29.190:35227/ using document root <none> and password file <none>
I20260925 04:55:55.728734 16596 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:55.728866 16596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:55.729128 16596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:55:55.732175 16596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "ce16c3437d3c46ed908e861e0fbaa8da"
format_stamp: "Formatted at 2026-09-25 04:55:55 on dist-test-slave-xzjr"
server_key: "bb985bb3b0b77b08d368122e5f3c2f27"
server_key_iv: "bf8049d8677d871595f594a0016a3477"
server_key_version: "encryptionkey@0"
I20260925 04:55:55.732884 16596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "ce16c3437d3c46ed908e861e0fbaa8da"
format_stamp: "Formatted at 2026-09-25 04:55:55 on dist-test-slave-xzjr"
server_key: "bb985bb3b0b77b08d368122e5f3c2f27"
server_key_iv: "bf8049d8677d871595f594a0016a3477"
server_key_version: "encryptionkey@0"
I20260925 04:55:55.738133 16596 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 04:55:55.741657 16611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:55.743046 16596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:55:55.743201 16596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "ce16c3437d3c46ed908e861e0fbaa8da"
format_stamp: "Formatted at 2026-09-25 04:55:55 on dist-test-slave-xzjr"
server_key: "bb985bb3b0b77b08d368122e5f3c2f27"
server_key_iv: "bf8049d8677d871595f594a0016a3477"
server_key_version: "encryptionkey@0"
I20260925 04:55:55.743366 16596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:55.755733 16596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:55:55.759716 16596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:55:55.760066 16596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:55:55.779565 16596 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:33109
I20260925 04:55:55.779597 16662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:33109 every 8 connection(s)
I20260925 04:55:55.781018 16596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:55:55.784770 16663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:55:55.784901 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 16596
I20260925 04:55:55.785137 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 04:55:55.785852 6262 external_mini_cluster.cc:1428] Setting key 91b271999a9d5122f94238047516050d
I20260925 04:55:55.799428 16663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce16c3437d3c46ed908e861e0fbaa8da: Bootstrap starting.
Sep 25 04:55:55 dist-test-slave-xzjr krb5kdc[16583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:55:55.802675 16663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce16c3437d3c46ed908e861e0fbaa8da: Neither blocks nor log segments found. Creating new log.
I20260925 04:55:55.804118 16663 log.cc:824] T 00000000000000000000000000000000 P ce16c3437d3c46ed908e861e0fbaa8da: Log is configured to *not* fsync() on all Append() calls
I20260925 04:55:55.807381 16663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce16c3437d3c46ed908e861e0fbaa8da: No bootstrap required, opened a new log
I20260925 04:55:55.810958 16666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:55.787875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48688 (local address 127.6.29.190:33109)
0925 04:55:55.788707 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:55.788727 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:55.788782 (+ 55us) server_negotiation.cc:407] Connection header received
0925 04:55:55.791533 (+ 2751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:55.791562 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:55.792140 (+ 578us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:55.792587 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:55.793909 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:55.795976 (+ 2067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:55.797212 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:55.798252 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:55.798440 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:55.802355 (+ 3915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:55.802365 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:55.802386 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:55.802436 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:55.806275 (+ 3839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:55.807011 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:55.807018 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:55.807027 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:55.807222 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:55.807640 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:55.807648 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:55.807653 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:55.808399 (+ 746us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:55.808717 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:55.809101 (+ 384us) server_negotiation.cc:299] Negotiation successful
0925 04:55:55.809550 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":356,"threads_started":1}
I20260925 04:55:55.816641 16663 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce16c3437d3c46ed908e861e0fbaa8da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce16c3437d3c46ed908e861e0fbaa8da" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33109 } }
I20260925 04:55:55.816990 16663 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce16c3437d3c46ed908e861e0fbaa8da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:55:55.817063 16663 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce16c3437d3c46ed908e861e0fbaa8da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce16c3437d3c46ed908e861e0fbaa8da, State: Initialized, Role: FOLLOWER
I20260925 04:55:55.817607 16663 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce16c3437d3c46ed908e861e0fbaa8da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce16c3437d3c46ed908e861e0fbaa8da" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33109 } }
I20260925 04:55:55.817796 16663 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce16c3437d3c46ed908e861e0fbaa8da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:55:55.817888 16663 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce16c3437d3c46ed908e861e0fbaa8da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:55:55.818008 16663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce16c3437d3c46ed908e861e0fbaa8da [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:55:55.820401 16663 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce16c3437d3c46ed908e861e0fbaa8da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce16c3437d3c46ed908e861e0fbaa8da" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33109 } }
I20260925 04:55:55.820901 16663 leader_election.cc:302] T 00000000000000000000000000000000 P ce16c3437d3c46ed908e861e0fbaa8da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce16c3437d3c46ed908e861e0fbaa8da; no voters:
I20260925 04:55:55.821411 16663 leader_election.cc:288] T 00000000000000000000000000000000 P ce16c3437d3c46ed908e861e0fbaa8da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:55:55.821552 16668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce16c3437d3c46ed908e861e0fbaa8da [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:55:55.822256 16668 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce16c3437d3c46ed908e861e0fbaa8da [term 1 LEADER]: Becoming Leader. State: Replica: ce16c3437d3c46ed908e861e0fbaa8da, State: Running, Role: LEADER
I20260925 04:55:55.822803 16668 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce16c3437d3c46ed908e861e0fbaa8da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce16c3437d3c46ed908e861e0fbaa8da" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33109 } }
I20260925 04:55:55.823496 16663 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce16c3437d3c46ed908e861e0fbaa8da [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:55:55.825603 16669 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce16c3437d3c46ed908e861e0fbaa8da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce16c3437d3c46ed908e861e0fbaa8da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce16c3437d3c46ed908e861e0fbaa8da" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33109 } } }
I20260925 04:55:55.825933 16669 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce16c3437d3c46ed908e861e0fbaa8da [sys.catalog]: This master's current role is: LEADER
I20260925 04:55:55.826208 16670 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce16c3437d3c46ed908e861e0fbaa8da [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce16c3437d3c46ed908e861e0fbaa8da. Latest consensus state: current_term: 1 leader_uuid: "ce16c3437d3c46ed908e861e0fbaa8da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce16c3437d3c46ed908e861e0fbaa8da" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33109 } } }
I20260925 04:55:55.827208 16670 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce16c3437d3c46ed908e861e0fbaa8da [sys.catalog]: This master's current role is: LEADER
I20260925 04:55:55.828370 16673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:55:55.834102 16673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:55:55.844296 16673 catalog_manager.cc:1347] Generated new cluster ID: 38e1ee0f223c483cb2ae8e845cbb61ca
I20260925 04:55:55.844420 16673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:55:55.867292 16673 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:55:55.869153 16673 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:55:55.876833 16673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce16c3437d3c46ed908e861e0fbaa8da: Generated new TSK 0
I20260925 04:55:55.877663 16673 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.29.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab.
Entry for principal kudu/127.6.29.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab.
I20260925 04:55:55.943518 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.129:0
--local_ip_for_outbound_sockets=127.6.29.129
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:33109
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:34737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:55:56.301929 16689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:55:56.302237 16689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:55:56.302412 16689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:55:56.309906 16689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:55:56.310027 16689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:55:56.310227 16689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.129
I20260925 04:55:56.321524 16689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:34737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:33109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.129
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:55:56.322990 16689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:55:56.324582 16689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:55:56.338472 16695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:56.338572 16694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:56.338538 16697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:55:56.339560 16689 server_base.cc:1034] running on GCE node
I20260925 04:55:56.340247 16689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:55:56.341357 16689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:55:56.342619 16689 hybrid_clock.cc:648] HybridClock initialized: now 1790312156342556 us; error 62 us; skew 500 ppm
I20260925 04:55:56.342900 16689 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:55:56 dist-test-slave-xzjr krb5kdc[16583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312156, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:55:56.346217 16689 init.cc:377] Logged in from keytab as kudu/127.6.29.129@KRBTEST.COM (short username kudu)
I20260925 04:55:56.347997 16689 webserver.cc:460] Webserver started at http://127.6.29.129:36191/ using document root <none> and password file <none>
I20260925 04:55:56.348805 16689 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:56.348955 16689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:56.349267 16689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:55:56.352311 16689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/data/instance:
uuid: "898ebeecef974fbfa298d726ebd56bf3"
format_stamp: "Formatted at 2026-09-25 04:55:56 on dist-test-slave-xzjr"
server_key: "5962154f6eb58e9c6713bc582fbb466e"
server_key_iv: "de624c27e5d5767880c561231f801d7f"
server_key_version: "encryptionkey@0"
I20260925 04:55:56.353116 16689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance:
uuid: "898ebeecef974fbfa298d726ebd56bf3"
format_stamp: "Formatted at 2026-09-25 04:55:56 on dist-test-slave-xzjr"
server_key: "5962154f6eb58e9c6713bc582fbb466e"
server_key_iv: "de624c27e5d5767880c561231f801d7f"
server_key_version: "encryptionkey@0"
I20260925 04:55:56.358642 16689 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 04:55:56.362497 16704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:56.364076 16689 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 04:55:56.364271 16689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/wal
uuid: "898ebeecef974fbfa298d726ebd56bf3"
format_stamp: "Formatted at 2026-09-25 04:55:56 on dist-test-slave-xzjr"
server_key: "5962154f6eb58e9c6713bc582fbb466e"
server_key_iv: "de624c27e5d5767880c561231f801d7f"
server_key_version: "encryptionkey@0"
I20260925 04:55:56.364486 16689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:56.377905 16689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:55:56.382022 16689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:55:56.382309 16689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:55:56.383232 16689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:55:56.384639 16689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:55:56.384742 16689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:56.384884 16689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:55:56.384954 16689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:56.418571 16689 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.129:34169
I20260925 04:55:56.418674 16816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.129:34169 every 8 connection(s)
I20260925 04:55:56.419957 16689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
I20260925 04:55:56.420349 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 16689
I20260925 04:55:56.420548 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance
I20260925 04:55:56.421399 6262 external_mini_cluster.cc:1428] Setting key 73483f65449fa4b64d39967205916c44
Sep 25 04:55:56 dist-test-slave-xzjr krb5kdc[16583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312156, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:55:56.442876 16819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:56.422553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:41301 (local address 127.6.29.190:33109)
0925 04:55:56.423024 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:56.423030 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:56.424534 (+ 1504us) server_negotiation.cc:407] Connection header received
0925 04:55:56.426442 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:56.426450 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:56.426542 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:56.426663 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:56.428298 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:56.429463 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:56.431344 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:56.431979 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:56.432087 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:56.436765 (+ 4678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:56.436770 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:56.436774 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:56.436808 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:56.439398 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:56.440398 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:56.440403 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:56.440405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:56.440521 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:56.441024 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:56.441027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:56.441029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:56.441325 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:56.441457 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:56.442439 (+ 982us) server_negotiation.cc:299] Negotiation successful
0925 04:55:56.442621 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":245,"threads_started":1}
I20260925 04:55:56.444648 16820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:56.423176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:33109 (local address 127.6.29.129:41301)
0925 04:55:56.424367 (+ 1191us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:56.424421 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:56.426164 (+ 1743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:56.426866 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:56.426874 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:56.427419 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:56.428116 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:56.428135 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:56.429651 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:56.429656 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:56.431172 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:56.431184 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:56.432213 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:56.432223 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:56.432573 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:56.433411 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:56.433445 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:56.436508 (+ 3063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:56.439551 (+ 3043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:56.439563 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:56.439585 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:56.440272 (+ 687us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:56.440631 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:56.440640 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:56.440646 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:56.440905 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:56.441483 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:56.441493 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:56.442009 (+ 516us) client_negotiation.cc:769] Sending connection context
0925 04:55:56.442347 (+ 338us) client_negotiation.cc:240] Negotiation successful
0925 04:55:56.442959 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875,"thread_start_us":649,"threads_started":1}
I20260925 04:55:56.446353 16817 heartbeater.cc:347] Connected to a master server at 127.6.29.190:33109
I20260925 04:55:56.446678 16817 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.6.29.130@KRBTEST.COM; defaulting to no policy
I20260925 04:55:56.447512 16817 heartbeater.cc:511] Master 127.6.29.190:33109 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.130@KRBTEST.COM" created.
I20260925 04:55:56.449829 16628 ts_manager.cc:194] Registered new tserver with Master: 898ebeecef974fbfa298d726ebd56bf3 (127.6.29.129:34169)
I20260925 04:55:56.452051 16628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.129@KRBTEST.COM'} at 127.6.29.129:41301
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab.
Entry for principal kudu/127.6.29.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab.
I20260925 04:55:56.480140 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.130:0
--local_ip_for_outbound_sockets=127.6.29.130
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:33109
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:34737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:55:56.818437 16823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:55:56.818742 16823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:55:56.818845 16823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:55:56.826301 16823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:55:56.826424 16823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:55:56.826539 16823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.130
I20260925 04:55:56.836539 16823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:34737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:33109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.130
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:55:56.837858 16823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:55:56.839131 16823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:55:56.852247 16828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:56.852439 16829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:56.852831 16831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:55:56.853108 16823 server_base.cc:1034] running on GCE node
I20260925 04:55:56.853789 16823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:55:56.854764 16823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:55:56.855984 16823 hybrid_clock.cc:648] HybridClock initialized: now 1790312156855939 us; error 60 us; skew 500 ppm
I20260925 04:55:56.856297 16823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:55:56 dist-test-slave-xzjr krb5kdc[16583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312156, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:55:56.859452 16823 init.cc:377] Logged in from keytab as kudu/127.6.29.130@KRBTEST.COM (short username kudu)
I20260925 04:55:56.861104 16823 webserver.cc:460] Webserver started at http://127.6.29.130:41861/ using document root <none> and password file <none>
I20260925 04:55:56.861807 16823 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:56.861958 16823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:56.862265 16823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:55:56.865262 16823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/data/instance:
uuid: "0cc33a5ecb144bed985ef27de338db63"
format_stamp: "Formatted at 2026-09-25 04:55:56 on dist-test-slave-xzjr"
server_key: "d710f9705ecc2c62515d83703ff521bd"
server_key_iv: "bb4149fe274e95197208f43c6ae49f97"
server_key_version: "encryptionkey@0"
I20260925 04:55:56.866058 16823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance:
uuid: "0cc33a5ecb144bed985ef27de338db63"
format_stamp: "Formatted at 2026-09-25 04:55:56 on dist-test-slave-xzjr"
server_key: "d710f9705ecc2c62515d83703ff521bd"
server_key_iv: "bb4149fe274e95197208f43c6ae49f97"
server_key_version: "encryptionkey@0"
I20260925 04:55:56.871410 16823 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 04:55:56.874945 16838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:56.876353 16823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 04:55:56.876533 16823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/wal
uuid: "0cc33a5ecb144bed985ef27de338db63"
format_stamp: "Formatted at 2026-09-25 04:55:56 on dist-test-slave-xzjr"
server_key: "d710f9705ecc2c62515d83703ff521bd"
server_key_iv: "bb4149fe274e95197208f43c6ae49f97"
server_key_version: "encryptionkey@0"
I20260925 04:55:56.876747 16823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:56.904181 16823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:55:56.908663 16823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:55:56.908977 16823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:55:56.909972 16823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:55:56.911412 16823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:55:56.911527 16823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:56.911682 16823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:55:56.911758 16823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:56.946398 16823 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.130:46869
I20260925 04:55:56.946456 16950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.130:46869 every 8 connection(s)
I20260925 04:55:56.947902 16823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
I20260925 04:55:56.954063 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 16823
I20260925 04:55:56.954281 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance
I20260925 04:55:56.954773 6262 external_mini_cluster.cc:1428] Setting key fd3ad35a74e606487b77a95a15df0b97
Sep 25 04:55:56 dist-test-slave-xzjr krb5kdc[16583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312156, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:55:56.968686 16953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:56.950624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:50485 (local address 127.6.29.190:33109)
0925 04:55:56.951165 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:56.951169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:56.952244 (+ 1075us) server_negotiation.cc:407] Connection header received
0925 04:55:56.954186 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:56.954192 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:56.954287 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:56.954411 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:56.956190 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:56.957094 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:56.958654 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:56.959098 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:56.959232 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:56.962657 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:56.962662 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:56.962665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:56.962695 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:56.965338 (+ 2643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:56.966248 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:56.966252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:56.966254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:56.966380 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:56.966874 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:56.966877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:56.966879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:56.967120 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:56.967224 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:56.968148 (+ 924us) server_negotiation.cc:299] Negotiation successful
0925 04:55:56.968371 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":326,"threads_started":1}
I20260925 04:55:56.970191 16954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:56.951303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:33109 (local address 127.6.29.130:50485)
0925 04:55:56.952076 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:56.952119 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:56.953905 (+ 1786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:56.954662 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:56.954671 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:56.955239 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:56.956024 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:56.956043 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:56.957238 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:56.957249 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:56.958503 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:56.958514 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:56.959328 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:56.959336 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:56.959568 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:56.960314 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:56.960336 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:56.962419 (+ 2083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:56.965487 (+ 3068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:56.965499 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:56.965519 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:56.966128 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:56.966509 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:56.966517 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:56.966523 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:56.966762 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:56.967241 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:56.967251 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:56.967740 (+ 489us) client_negotiation.cc:769] Sending connection context
0925 04:55:56.968063 (+ 323us) client_negotiation.cc:240] Negotiation successful
0925 04:55:56.968692 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":270,"threads_started":1}
I20260925 04:55:56.971803 16951 heartbeater.cc:347] Connected to a master server at 127.6.29.190:33109
I20260925 04:55:56.972153 16951 heartbeater.cc:464] Registering TS with master...
I20260925 04:55:56.972875 16951 heartbeater.cc:511] Master 127.6.29.190:33109 requested a full tablet report, sending...
I20260925 04:55:56.974489 16628 ts_manager.cc:194] Registered new tserver with Master: 0cc33a5ecb144bed985ef27de338db63 (127.6.29.130:46869)
I20260925 04:55:56.975322 16628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.130@KRBTEST.COM'} at 127.6.29.130:50485
WARNING: no policy specified for kudu/127.6.29.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab.
Entry for principal kudu/127.6.29.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab.
I20260925 04:55:57.007880 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.131:0
--local_ip_for_outbound_sockets=127.6.29.131
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:33109
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:34737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:55:57.342128 16957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:55:57.342446 16957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:55:57.342617 16957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:55:57.350219 16957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:55:57.350374 16957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:55:57.350553 16957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.131
I20260925 04:55:57.361595 16957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:34737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:33109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.131
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:55:57.363037 16957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:55:57.364543 16957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:55:57.378046 16965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:57.378152 16963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:55:57.378115 16962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:55:57.379524 16957 server_base.cc:1034] running on GCE node
I20260925 04:55:57.380265 16957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:55:57.381320 16957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:55:57.382535 16957 hybrid_clock.cc:648] HybridClock initialized: now 1790312157382461 us; error 59 us; skew 500 ppm
I20260925 04:55:57.382875 16957 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:55:57 dist-test-slave-xzjr krb5kdc[16583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312157, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:55:57.386087 16957 init.cc:377] Logged in from keytab as kudu/127.6.29.131@KRBTEST.COM (short username kudu)
I20260925 04:55:57.387562 16957 webserver.cc:460] Webserver started at http://127.6.29.131:38595/ using document root <none> and password file <none>
I20260925 04:55:57.388294 16957 fs_manager.cc:359] Metadata directory not provided
I20260925 04:55:57.388394 16957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:55:57.388631 16957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:55:57.391579 16957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/data/instance:
uuid: "8617942027c7462686da65ab7917564a"
format_stamp: "Formatted at 2026-09-25 04:55:57 on dist-test-slave-xzjr"
server_key: "1b8fbb031ba115f70d5d898f5f64095f"
server_key_iv: "259aee2dfb608eee1aedd79fe382b890"
server_key_version: "encryptionkey@0"
I20260925 04:55:57.392432 16957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance:
uuid: "8617942027c7462686da65ab7917564a"
format_stamp: "Formatted at 2026-09-25 04:55:57 on dist-test-slave-xzjr"
server_key: "1b8fbb031ba115f70d5d898f5f64095f"
server_key_iv: "259aee2dfb608eee1aedd79fe382b890"
server_key_version: "encryptionkey@0"
I20260925 04:55:57.397692 16957 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 04:55:57.401356 16972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:57.402783 16957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260925 04:55:57.402963 16957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/wal
uuid: "8617942027c7462686da65ab7917564a"
format_stamp: "Formatted at 2026-09-25 04:55:57 on dist-test-slave-xzjr"
server_key: "1b8fbb031ba115f70d5d898f5f64095f"
server_key_iv: "259aee2dfb608eee1aedd79fe382b890"
server_key_version: "encryptionkey@0"
I20260925 04:55:57.403172 16957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:55:57.416452 16957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:55:57.421546 16957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:55:57.421873 16957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:55:57.423028 16957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:55:57.424957 16957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:55:57.425069 16957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:57.425190 16957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:55:57.425259 16957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:55:57.455255 16817 heartbeater.cc:503] Master 127.6.29.190:33109 was elected leader, sending a full tablet report...
I20260925 04:55:57.460191 16957 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.131:39243
I20260925 04:55:57.460238 17084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.131:39243 every 8 connection(s)
I20260925 04:55:57.461552 16957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
I20260925 04:55:57.471693 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 16957
I20260925 04:55:57.471906 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance
I20260925 04:55:57.472468 6262 external_mini_cluster.cc:1428] Setting key 31a59129318b3fdd2777a3a5754e2375
Sep 25 04:55:57 dist-test-slave-xzjr krb5kdc[16583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312157, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:55:57.483700 16953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:57.464345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:35355 (local address 127.6.29.190:33109)
0925 04:55:57.464520 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:57.464524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:57.465717 (+ 1193us) server_negotiation.cc:407] Connection header received
0925 04:55:57.467703 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:57.467709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:57.467835 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:57.467952 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:57.469648 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:57.470471 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:57.472693 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:57.473104 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:57.473252 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:57.477448 (+ 4196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:57.477454 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:57.477459 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:57.477500 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:57.480391 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:57.481296 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:57.481302 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:57.481306 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:57.481443 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:57.481909 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:57.481915 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:57.481919 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:57.482203 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:57.482379 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:57.483112 (+ 733us) server_negotiation.cc:299] Negotiation successful
0925 04:55:57.483346 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 04:55:57.484789 17087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:57.464828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:33109 (local address 127.6.29.131:35355)
0925 04:55:57.465539 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:57.465584 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:57.467385 (+ 1801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:57.468208 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:57.468217 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:57.468871 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:55:57.469455 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:57.469474 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:57.470661 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:57.470678 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:57.472496 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:57.472515 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:57.473338 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:57.473344 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:57.473598 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:57.474514 (+ 916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:55:57.474547 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:55:57.477233 (+ 2686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:55:57.480555 (+ 3322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:57.480565 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:57.480587 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:57.481166 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:57.481565 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:55:57.481572 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:55:57.481576 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:55:57.481790 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:55:57.482342 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:55:57.482352 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:55:57.482743 (+ 391us) client_negotiation.cc:769] Sending connection context
0925 04:55:57.483026 (+ 283us) client_negotiation.cc:240] Negotiation successful
0925 04:55:57.483595 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":296,"threads_started":1}
I20260925 04:55:57.486275 17085 heartbeater.cc:347] Connected to a master server at 127.6.29.190:33109
I20260925 04:55:57.486649 17085 heartbeater.cc:464] Registering TS with master...
I20260925 04:55:57.487547 17085 heartbeater.cc:511] Master 127.6.29.190:33109 requested a full tablet report, sending...
I20260925 04:55:57.489231 16628 ts_manager.cc:194] Registered new tserver with Master: 8617942027c7462686da65ab7917564a (127.6.29.131:39243)
I20260925 04:55:57.490051 16628 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.131@KRBTEST.COM'} at 127.6.29.131:35355
I20260925 04:55:57.501333 6262 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:55:57.519047 16953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:57.506383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48700 (local address 127.6.29.190:33109)
0925 04:55:57.506548 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:57.506553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:57.507131 (+ 578us) server_negotiation.cc:407] Connection header received
0925 04:55:57.507220 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:57.507224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:57.507303 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:57.507382 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:57.508989 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:57.509764 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:57.510992 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:57.511394 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:57.511496 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:57.513766 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:57.513771 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:57.513774 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:57.513808 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:57.516369 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:57.517256 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:57.517260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:57.517262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:57.517383 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:57.517738 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:57.517741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:57.517748 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:57.518091 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:57.518289 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:57.518601 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 04:55:57.518786 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 04:55:57.523734 16628 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:55:57.531097 16628 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:55:57.557567 17102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:57.546631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37872 (local address 127.6.29.130:46869)
0925 04:55:57.549071 (+ 2440us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:57.549084 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:57.549661 (+ 577us) server_negotiation.cc:407] Connection header received
0925 04:55:57.550030 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:57.550038 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:57.550276 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:57.550441 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:55:57.551434 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:57.552705 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:57.554784 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:57.556394 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:57.556563 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:57.556785 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:57.557055 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 04:55:57.557229 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2249,"thread_start_us":2148,"threads_started":1}
I20260925 04:55:57.557756 17101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:57.546095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:46869 (local address 127.0.0.1:37872)
0925 04:55:57.549535 (+ 3440us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:57.549556 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:57.549826 (+ 270us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:57.550721 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:57.550726 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:57.550753 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:55:57.551249 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:57.551260 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:57.552853 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:57.552857 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:57.554624 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:57.554636 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:57.556598 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:57.556604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:57.556795 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:57.556857 (+ 62us) client_negotiation.cc:769] Sending connection context
0925 04:55:57.557010 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 04:55:57.557206 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3322,"mutex_wait_us":2,"thread_start_us":3177,"threads_started":1}
I20260925 04:55:57.558108 17097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:57.543570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.131:39243 (local address 127.0.0.1:53168)
0925 04:55:57.545728 (+ 2158us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:57.545779 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:57.546027 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:57.550522 (+ 4495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:57.550528 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:57.550555 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:55:57.551038 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:57.551056 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:57.552976 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:57.552981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:57.555333 (+ 2352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:57.555344 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:57.557326 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:57.557335 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:57.557524 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:57.557557 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 04:55:57.557631 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 04:55:57.557759 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1798,"thread_start_us":1615,"threads_started":1}
I20260925 04:55:57.558688 17098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:57.543991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53168 (local address 127.6.29.131:39243)
0925 04:55:57.549958 (+ 5967us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:57.549966 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:57.549995 (+ 29us) server_negotiation.cc:407] Connection header received
0925 04:55:57.550079 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:57.550085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:57.550275 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:57.550572 (+ 297us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:55:57.551293 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:57.552840 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:57.555614 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:57.557099 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:57.557890 (+ 791us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:57.558092 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:57.558150 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 04:55:57.558344 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5796,"thread_start_us":5647,"threads_started":1}
I20260925 04:55:57.559289 17099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:57.544416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:34169 (local address 127.0.0.1:53000)
0925 04:55:57.548776 (+ 4360us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:57.548797 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:57.548952 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:57.549941 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:57.549950 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:57.549985 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:55:57.550284 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:57.550293 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:57.552509 (+ 2216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:57.552514 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:57.555165 (+ 2651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:57.555181 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:57.558535 (+ 3354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:57.558540 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:57.558675 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:57.558695 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 04:55:57.558802 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 04:55:57.558933 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4195,"mutex_wait_us":1,"thread_start_us":4025,"threads_started":1}
I20260925 04:55:57.560834 17100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:57.545095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53000 (local address 127.6.29.129:34169)
0925 04:55:57.548249 (+ 3154us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:57.548264 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:57.549110 (+ 846us) server_negotiation.cc:407] Connection header received
0925 04:55:57.549244 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:57.549257 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:57.549598 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:57.549796 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:55:57.550481 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:57.552366 (+ 1885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:57.555368 (+ 3002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:57.557312 (+ 1944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:57.557439 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:57.557671 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:57.558999 (+ 1328us) server_negotiation.cc:299] Negotiation successful
0925 04:55:57.560503 (+ 1504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1596,"thread_start_us":1396,"threads_started":1}
I20260925 04:55:57.562069 17020 tablet_service.cc:1467] Processing CreateTablet for tablet 5a6d1d7d0572439aafb46eec67d3366b (DEFAULT_TABLE table=test-table [id=205a88f0a2684c7bb8cf1e8ed02ba675]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:55:57.562070 16886 tablet_service.cc:1467] Processing CreateTablet for tablet 5a6d1d7d0572439aafb46eec67d3366b (DEFAULT_TABLE table=test-table [id=205a88f0a2684c7bb8cf1e8ed02ba675]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:55:57.563253 16886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a6d1d7d0572439aafb46eec67d3366b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:55:57.563359 17020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a6d1d7d0572439aafb46eec67d3366b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:55:57.565469 16752 tablet_service.cc:1467] Processing CreateTablet for tablet 5a6d1d7d0572439aafb46eec67d3366b (DEFAULT_TABLE table=test-table [id=205a88f0a2684c7bb8cf1e8ed02ba675]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:55:57.566751 16752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a6d1d7d0572439aafb46eec67d3366b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:55:57.575397 17103 tablet_bootstrap.cc:492] T 5a6d1d7d0572439aafb46eec67d3366b P 8617942027c7462686da65ab7917564a: Bootstrap starting.
I20260925 04:55:57.578621 17103 tablet_bootstrap.cc:654] T 5a6d1d7d0572439aafb46eec67d3366b P 8617942027c7462686da65ab7917564a: Neither blocks nor log segments found. Creating new log.
I20260925 04:55:57.579730 17103 log.cc:824] T 5a6d1d7d0572439aafb46eec67d3366b P 8617942027c7462686da65ab7917564a: Log is configured to *not* fsync() on all Append() calls
I20260925 04:55:57.581411 17106 tablet_bootstrap.cc:492] T 5a6d1d7d0572439aafb46eec67d3366b P 0cc33a5ecb144bed985ef27de338db63: Bootstrap starting.
I20260925 04:55:57.583237 17103 tablet_bootstrap.cc:492] T 5a6d1d7d0572439aafb46eec67d3366b P 8617942027c7462686da65ab7917564a: No bootstrap required, opened a new log
I20260925 04:55:57.583523 17103 ts_tablet_manager.cc:1397] T 5a6d1d7d0572439aafb46eec67d3366b P 8617942027c7462686da65ab7917564a: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260925 04:55:57.584280 17105 tablet_bootstrap.cc:492] T 5a6d1d7d0572439aafb46eec67d3366b P 898ebeecef974fbfa298d726ebd56bf3: Bootstrap starting.
I20260925 04:55:57.586220 17106 tablet_bootstrap.cc:654] T 5a6d1d7d0572439aafb46eec67d3366b P 0cc33a5ecb144bed985ef27de338db63: Neither blocks nor log segments found. Creating new log.
I20260925 04:55:57.587085 17105 tablet_bootstrap.cc:654] T 5a6d1d7d0572439aafb46eec67d3366b P 898ebeecef974fbfa298d726ebd56bf3: Neither blocks nor log segments found. Creating new log.
I20260925 04:55:57.587522 17106 log.cc:824] T 5a6d1d7d0572439aafb46eec67d3366b P 0cc33a5ecb144bed985ef27de338db63: Log is configured to *not* fsync() on all Append() calls
I20260925 04:55:57.588253 17105 log.cc:824] T 5a6d1d7d0572439aafb46eec67d3366b P 898ebeecef974fbfa298d726ebd56bf3: Log is configured to *not* fsync() on all Append() calls
I20260925 04:55:57.590801 17106 tablet_bootstrap.cc:492] T 5a6d1d7d0572439aafb46eec67d3366b P 0cc33a5ecb144bed985ef27de338db63: No bootstrap required, opened a new log
I20260925 04:55:57.591090 17106 ts_tablet_manager.cc:1397] T 5a6d1d7d0572439aafb46eec67d3366b P 0cc33a5ecb144bed985ef27de338db63: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260925 04:55:57.593063 17105 tablet_bootstrap.cc:492] T 5a6d1d7d0572439aafb46eec67d3366b P 898ebeecef974fbfa298d726ebd56bf3: No bootstrap required, opened a new log
I20260925 04:55:57.593442 17105 ts_tablet_manager.cc:1397] T 5a6d1d7d0572439aafb46eec67d3366b P 898ebeecef974fbfa298d726ebd56bf3: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.002s
I20260925 04:55:57.594122 17103 raft_consensus.cc:348] T 5a6d1d7d0572439aafb46eec67d3366b P 8617942027c7462686da65ab7917564a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8617942027c7462686da65ab7917564a" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 39243 } } peers { permanent_uuid: "0cc33a5ecb144bed985ef27de338db63" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 46869 } } peers { permanent_uuid: "898ebeecef974fbfa298d726ebd56bf3" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 34169 } }
I20260925 04:55:57.594578 17103 raft_consensus.cc:374] T 5a6d1d7d0572439aafb46eec67d3366b P 8617942027c7462686da65ab7917564a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:55:57.594676 17103 raft_consensus.cc:729] T 5a6d1d7d0572439aafb46eec67d3366b P 8617942027c7462686da65ab7917564a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8617942027c7462686da65ab7917564a, State: Initialized, Role: FOLLOWER
I20260925 04:55:57.595409 17103 consensus_queue.cc:260] T 5a6d1d7d0572439aafb46eec67d3366b P 8617942027c7462686da65ab7917564a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8617942027c7462686da65ab7917564a" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 39243 } } peers { permanent_uuid: "0cc33a5ecb144bed985ef27de338db63" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 46869 } } peers { permanent_uuid: "898ebeecef974fbfa298d726ebd56bf3" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 34169 } }
I20260925 04:55:57.596554 17085 heartbeater.cc:503] Master 127.6.29.190:33109 was elected leader, sending a full tablet report...
I20260925 04:55:57.597227 17103 ts_tablet_manager.cc:1428] T 5a6d1d7d0572439aafb46eec67d3366b P 8617942027c7462686da65ab7917564a: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260925 04:55:57.601208 17106 raft_consensus.cc:348] T 5a6d1d7d0572439aafb46eec67d3366b P 0cc33a5ecb144bed985ef27de338db63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8617942027c7462686da65ab7917564a" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 39243 } } peers { permanent_uuid: "0cc33a5ecb144bed985ef27de338db63" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 46869 } } peers { permanent_uuid: "898ebeecef974fbfa298d726ebd56bf3" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 34169 } }
I20260925 04:55:57.601605 17106 raft_consensus.cc:374] T 5a6d1d7d0572439aafb46eec67d3366b P 0cc33a5ecb144bed985ef27de338db63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:55:57.601733 17106 raft_consensus.cc:729] T 5a6d1d7d0572439aafb46eec67d3366b P 0cc33a5ecb144bed985ef27de338db63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cc33a5ecb144bed985ef27de338db63, State: Initialized, Role: FOLLOWER
I20260925 04:55:57.602386 17106 consensus_queue.cc:260] T 5a6d1d7d0572439aafb46eec67d3366b P 0cc33a5ecb144bed985ef27de338db63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8617942027c7462686da65ab7917564a" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 39243 } } peers { permanent_uuid: "0cc33a5ecb144bed985ef27de338db63" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 46869 } } peers { permanent_uuid: "898ebeecef974fbfa298d726ebd56bf3" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 34169 } }
I20260925 04:55:57.603591 16951 heartbeater.cc:503] Master 127.6.29.190:33109 was elected leader, sending a full tablet report...
I20260925 04:55:57.603380 17105 raft_consensus.cc:348] T 5a6d1d7d0572439aafb46eec67d3366b P 898ebeecef974fbfa298d726ebd56bf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8617942027c7462686da65ab7917564a" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 39243 } } peers { permanent_uuid: "0cc33a5ecb144bed985ef27de338db63" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 46869 } } peers { permanent_uuid: "898ebeecef974fbfa298d726ebd56bf3" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 34169 } }
I20260925 04:55:57.603896 17105 raft_consensus.cc:374] T 5a6d1d7d0572439aafb46eec67d3366b P 898ebeecef974fbfa298d726ebd56bf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:55:57.604014 17105 raft_consensus.cc:729] T 5a6d1d7d0572439aafb46eec67d3366b P 898ebeecef974fbfa298d726ebd56bf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 898ebeecef974fbfa298d726ebd56bf3, State: Initialized, Role: FOLLOWER
I20260925 04:55:57.604466 17106 ts_tablet_manager.cc:1428] T 5a6d1d7d0572439aafb46eec67d3366b P 0cc33a5ecb144bed985ef27de338db63: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260925 04:55:57.604630 17105 consensus_queue.cc:260] T 5a6d1d7d0572439aafb46eec67d3366b P 898ebeecef974fbfa298d726ebd56bf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8617942027c7462686da65ab7917564a" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 39243 } } peers { permanent_uuid: "0cc33a5ecb144bed985ef27de338db63" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 46869 } } peers { permanent_uuid: "898ebeecef974fbfa298d726ebd56bf3" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 34169 } }
I20260925 04:55:57.606459 17105 ts_tablet_manager.cc:1428] T 5a6d1d7d0572439aafb46eec67d3366b P 898ebeecef974fbfa298d726ebd56bf3: Time spent starting tablet: real 0.013s user 0.010s sys 0.000s
W20260925 04:55:57.672472 16818 tablet.cc:2369] T 5a6d1d7d0572439aafb46eec67d3366b P 898ebeecef974fbfa298d726ebd56bf3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:55:57.700044 16952 tablet.cc:2369] T 5a6d1d7d0572439aafb46eec67d3366b P 0cc33a5ecb144bed985ef27de338db63: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:55:57.713318 17086 tablet.cc:2369] T 5a6d1d7d0572439aafb46eec67d3366b P 8617942027c7462686da65ab7917564a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:55:57.833410 17109 raft_consensus.cc:482] T 5a6d1d7d0572439aafb46eec67d3366b P 8617942027c7462686da65ab7917564a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:55:57.833793 17109 raft_consensus.cc:504] T 5a6d1d7d0572439aafb46eec67d3366b P 8617942027c7462686da65ab7917564a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8617942027c7462686da65ab7917564a" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 39243 } } peers { permanent_uuid: "0cc33a5ecb144bed985ef27de338db63" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 46869 } } peers { permanent_uuid: "898ebeecef974fbfa298d726ebd56bf3" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 34169 } }
I20260925 04:55:57.835672 17109 leader_election.cc:288] T 5a6d1d7d0572439aafb46eec67d3366b P 8617942027c7462686da65ab7917564a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cc33a5ecb144bed985ef27de338db63 (127.6.29.130:46869), 898ebeecef974fbfa298d726ebd56bf3 (127.6.29.129:34169)
I20260925 04:55:57.842566 17087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:57.836034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:46869 (local address 127.6.29.131:55621)
0925 04:55:57.836285 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:57.836307 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:57.836447 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:57.836917 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:57.836922 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:57.836961 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:55:57.837207 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:57.837219 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:57.838621 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:57.838630 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:57.840767 (+ 2137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:57.840781 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:57.842026 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:57.842030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:57.842158 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:57.842182 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 04:55:57.842254 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 04:55:57.842347 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":32}
I20260925 04:55:57.842729 17102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:57.836147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:55621 (local address 127.6.29.130:46869)
0925 04:55:57.836347 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:57.836353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:57.836436 (+ 83us) server_negotiation.cc:407] Connection header received
0925 04:55:57.836658 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:57.836665 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:57.836781 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:57.836891 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:55:57.837396 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:57.838449 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:57.840937 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:57.841870 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:57.842030 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:57.842103 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:57.842306 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 04:55:57.842440 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 04:55:57.843683 17100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:57.836123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:52771 (local address 127.6.29.129:34169)
0925 04:55:57.836314 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:57.836321 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:57.837507 (+ 1186us) server_negotiation.cc:407] Connection header received
0925 04:55:57.837675 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:57.837682 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:57.837757 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:57.837860 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:55:57.838426 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:57.839237 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:57.841574 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:57.842751 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:57.842912 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:57.842991 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:57.843309 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 04:55:57.843434 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 04:55:57.843772 17112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:57.836435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:34169 (local address 127.6.29.131:52771)
0925 04:55:57.837384 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0925 04:55:57.837401 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:55:57.837536 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:55:57.837911 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:55:57.837916 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:55:57.837939 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:55:57.838234 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:57.838247 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:57.839388 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:57.839393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:57.841419 (+ 2026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:55:57.841430 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:55:57.843032 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:55:57.843042 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:55:57.843152 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:57.843174 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:55:57.843284 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 04:55:57.843404 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"mutex_wait_us":1,"thread_start_us":413,"threads_started":1}
I20260925 04:55:57.843674 16906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a6d1d7d0572439aafb46eec67d3366b" candidate_uuid: "8617942027c7462686da65ab7917564a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cc33a5ecb144bed985ef27de338db63" is_pre_election: true
I20260925 04:55:57.844310 16906 raft_consensus.cc:2454] T 5a6d1d7d0572439aafb46eec67d3366b P 0cc33a5ecb144bed985ef27de338db63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8617942027c7462686da65ab7917564a in term 0.
I20260925 04:55:57.844898 16772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a6d1d7d0572439aafb46eec67d3366b" candidate_uuid: "8617942027c7462686da65ab7917564a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "898ebeecef974fbfa298d726ebd56bf3" is_pre_election: true
I20260925 04:55:57.845052 16976 leader_election.cc:302] T 5a6d1d7d0572439aafb46eec67d3366b P 8617942027c7462686da65ab7917564a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cc33a5ecb144bed985ef27de338db63, 8617942027c7462686da65ab7917564a; no voters:
I20260925 04:55:57.845394 16772 raft_consensus.cc:2454] T 5a6d1d7d0572439aafb46eec67d3366b P 898ebeecef974fbfa298d726ebd56bf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8617942027c7462686da65ab7917564a in term 0.
I20260925 04:55:57.845479 17109 raft_consensus.cc:2781] T 5a6d1d7d0572439aafb46eec67d3366b P 8617942027c7462686da65ab7917564a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:55:57.845616 17109 raft_consensus.cc:482] T 5a6d1d7d0572439aafb46eec67d3366b P 8617942027c7462686da65ab7917564a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:55:57.845728 17109 raft_consensus.cc:3037] T 5a6d1d7d0572439aafb46eec67d3366b P 8617942027c7462686da65ab7917564a [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:55:57.848309 17109 raft_consensus.cc:504] T 5a6d1d7d0572439aafb46eec67d3366b P 8617942027c7462686da65ab7917564a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8617942027c7462686da65ab7917564a" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 39243 } } peers { permanent_uuid: "0cc33a5ecb144bed985ef27de338db63" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 46869 } } peers { permanent_uuid: "898ebeecef974fbfa298d726ebd56bf3" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 34169 } }
I20260925 04:55:57.848858 17109 leader_election.cc:288] T 5a6d1d7d0572439aafb46eec67d3366b P 8617942027c7462686da65ab7917564a [CANDIDATE]: Term 1 election: Requested vote from peers 0cc33a5ecb144bed985ef27de338db63 (127.6.29.130:46869), 898ebeecef974fbfa298d726ebd56bf3 (127.6.29.129:34169)
I20260925 04:55:57.849261 16906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a6d1d7d0572439aafb46eec67d3366b" candidate_uuid: "8617942027c7462686da65ab7917564a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cc33a5ecb144bed985ef27de338db63"
I20260925 04:55:57.849354 16772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a6d1d7d0572439aafb46eec67d3366b" candidate_uuid: "8617942027c7462686da65ab7917564a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "898ebeecef974fbfa298d726ebd56bf3"
I20260925 04:55:57.849457 16906 raft_consensus.cc:3037] T 5a6d1d7d0572439aafb46eec67d3366b P 0cc33a5ecb144bed985ef27de338db63 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:55:57.849575 16772 raft_consensus.cc:3037] T 5a6d1d7d0572439aafb46eec67d3366b P 898ebeecef974fbfa298d726ebd56bf3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:55:57.851744 16772 raft_consensus.cc:2454] T 5a6d1d7d0572439aafb46eec67d3366b P 898ebeecef974fbfa298d726ebd56bf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8617942027c7462686da65ab7917564a in term 1.
I20260925 04:55:57.851752 16906 raft_consensus.cc:2454] T 5a6d1d7d0572439aafb46eec67d3366b P 0cc33a5ecb144bed985ef27de338db63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8617942027c7462686da65ab7917564a in term 1.
I20260925 04:55:57.852212 16976 leader_election.cc:302] T 5a6d1d7d0572439aafb46eec67d3366b P 8617942027c7462686da65ab7917564a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cc33a5ecb144bed985ef27de338db63, 8617942027c7462686da65ab7917564a; no voters:
I20260925 04:55:57.852545 17109 raft_consensus.cc:2781] T 5a6d1d7d0572439aafb46eec67d3366b P 8617942027c7462686da65ab7917564a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:55:57.852994 17109 raft_consensus.cc:686] T 5a6d1d7d0572439aafb46eec67d3366b P 8617942027c7462686da65ab7917564a [term 1 LEADER]: Becoming Leader. State: Replica: 8617942027c7462686da65ab7917564a, State: Running, Role: LEADER
I20260925 04:55:57.853477 17109 consensus_queue.cc:237] T 5a6d1d7d0572439aafb46eec67d3366b P 8617942027c7462686da65ab7917564a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8617942027c7462686da65ab7917564a" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 39243 } } peers { permanent_uuid: "0cc33a5ecb144bed985ef27de338db63" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 46869 } } peers { permanent_uuid: "898ebeecef974fbfa298d726ebd56bf3" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 34169 } }
I20260925 04:55:57.858489 16627 catalog_manager.cc:5495] T 5a6d1d7d0572439aafb46eec67d3366b P 8617942027c7462686da65ab7917564a reported cstate change: term changed from 0 to 1, leader changed from <none> to 8617942027c7462686da65ab7917564a (127.6.29.131). New cstate: current_term: 1 leader_uuid: "8617942027c7462686da65ab7917564a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8617942027c7462686da65ab7917564a" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 39243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cc33a5ecb144bed985ef27de338db63" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 46869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "898ebeecef974fbfa298d726ebd56bf3" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 34169 } health_report { overall_health: UNKNOWN } } }
I20260925 04:55:57.921427 17098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:57.916305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53182 (local address 127.6.29.131:39243)
0925 04:55:57.916528 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:57.916533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:57.916552 (+ 19us) server_negotiation.cc:407] Connection header received
0925 04:55:57.916769 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:57.916773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:57.916851 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:57.916977 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:55:57.917336 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:57.918127 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:57.919383 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:57.919878 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:57.919976 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:57.920411 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:55:57.920936 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:55:57.921052 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:57.921154 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 04:55:57.921223 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260925 04:55:57.929333 16772 raft_consensus.cc:1261] T 5a6d1d7d0572439aafb46eec67d3366b P 898ebeecef974fbfa298d726ebd56bf3 [term 1 FOLLOWER]: Refusing update from remote peer 8617942027c7462686da65ab7917564a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:55:57.929464 16906 raft_consensus.cc:1261] T 5a6d1d7d0572439aafb46eec67d3366b P 0cc33a5ecb144bed985ef27de338db63 [term 1 FOLLOWER]: Refusing update from remote peer 8617942027c7462686da65ab7917564a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:55:57.930424 17113 consensus_queue.cc:1035] T 5a6d1d7d0572439aafb46eec67d3366b P 8617942027c7462686da65ab7917564a [LEADER]: Connected to new peer: Peer: permanent_uuid: "898ebeecef974fbfa298d726ebd56bf3" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 34169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:55:57.930821 17109 consensus_queue.cc:1035] T 5a6d1d7d0572439aafb46eec67d3366b P 8617942027c7462686da65ab7917564a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cc33a5ecb144bed985ef27de338db63" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 46869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:55:57.942526 17116 mvcc.cc:205] Tried to move back new op lower bound from 7333118598868938752 to 7333118598571634688. Current Snapshot: MvccSnapshot[applied={T|T < 7333118598868938752}]
I20260925 04:55:57.945084 17118 mvcc.cc:205] Tried to move back new op lower bound from 7333118598868938752 to 7333118598571634688. Current Snapshot: MvccSnapshot[applied={T|T < 7333118598868938752}]
I20260925 04:55:57.957358 16628 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:55:57.957698 16628 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:55:57.961824 16628 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ce16c3437d3c46ed908e861e0fbaa8da: Sending DeleteTablet for 3 replicas of tablet 5a6d1d7d0572439aafb46eec67d3366b
I20260925 04:55:57.963322 16752 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a6d1d7d0572439aafb46eec67d3366b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:55:57 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:53000
I20260925 04:55:57.963423 17020 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a6d1d7d0572439aafb46eec67d3366b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:55:57 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:53168
I20260925 04:55:57.964074 16886 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a6d1d7d0572439aafb46eec67d3366b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:55:57 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:37872
I20260925 04:55:57.970062 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 16689
I20260925 04:55:57.972474 17127 tablet_replica.cc:330] T 5a6d1d7d0572439aafb46eec67d3366b P 0cc33a5ecb144bed985ef27de338db63: stopping tablet replica
I20260925 04:55:57.972987 17127 raft_consensus.cc:2229] T 5a6d1d7d0572439aafb46eec67d3366b P 0cc33a5ecb144bed985ef27de338db63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:55:57.973572 17127 raft_consensus.cc:2258] T 5a6d1d7d0572439aafb46eec67d3366b P 0cc33a5ecb144bed985ef27de338db63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:55:57.974514 17126 tablet_replica.cc:330] T 5a6d1d7d0572439aafb46eec67d3366b P 8617942027c7462686da65ab7917564a: stopping tablet replica
I20260925 04:55:57.974915 17126 raft_consensus.cc:2229] T 5a6d1d7d0572439aafb46eec67d3366b P 8617942027c7462686da65ab7917564a [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:55:57.975615 17126 raft_consensus.cc:2258] T 5a6d1d7d0572439aafb46eec67d3366b P 8617942027c7462686da65ab7917564a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:55:57.976348 17125 tablet_replica.cc:330] T 5a6d1d7d0572439aafb46eec67d3366b P 898ebeecef974fbfa298d726ebd56bf3: stopping tablet replica
I20260925 04:55:57.976750 17125 raft_consensus.cc:2229] T 5a6d1d7d0572439aafb46eec67d3366b P 898ebeecef974fbfa298d726ebd56bf3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:55:57.977238 17125 raft_consensus.cc:2258] T 5a6d1d7d0572439aafb46eec67d3366b P 898ebeecef974fbfa298d726ebd56bf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:55:57.977650 17127 ts_tablet_manager.cc:1905] T 5a6d1d7d0572439aafb46eec67d3366b P 0cc33a5ecb144bed985ef27de338db63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:55:57.977692 17126 ts_tablet_manager.cc:1905] T 5a6d1d7d0572439aafb46eec67d3366b P 8617942027c7462686da65ab7917564a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:55:57.979503 17125 ts_tablet_manager.cc:1905] T 5a6d1d7d0572439aafb46eec67d3366b P 898ebeecef974fbfa298d726ebd56bf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:55:57.985065 17127 ts_tablet_manager.cc:1918] T 5a6d1d7d0572439aafb46eec67d3366b P 0cc33a5ecb144bed985ef27de338db63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:55:57.985085 17126 ts_tablet_manager.cc:1918] T 5a6d1d7d0572439aafb46eec67d3366b P 8617942027c7462686da65ab7917564a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:55:57.985257 17127 log.cc:1196] T 5a6d1d7d0572439aafb46eec67d3366b P 0cc33a5ecb144bed985ef27de338db63: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-1/wal/wals/5a6d1d7d0572439aafb46eec67d3366b
I20260925 04:55:57.985270 17126 log.cc:1196] T 5a6d1d7d0572439aafb46eec67d3366b P 8617942027c7462686da65ab7917564a: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-2/wal/wals/5a6d1d7d0572439aafb46eec67d3366b
I20260925 04:55:57.985828 17127 ts_tablet_manager.cc:1939] T 5a6d1d7d0572439aafb46eec67d3366b P 0cc33a5ecb144bed985ef27de338db63: Deleting consensus metadata
I20260925 04:55:57.985836 17126 ts_tablet_manager.cc:1939] T 5a6d1d7d0572439aafb46eec67d3366b P 8617942027c7462686da65ab7917564a: Deleting consensus metadata
I20260925 04:55:57.987715 16615 catalog_manager.cc:4841] TS 8617942027c7462686da65ab7917564a (127.6.29.131:39243): tablet 5a6d1d7d0572439aafb46eec67d3366b (table test-table [id=205a88f0a2684c7bb8cf1e8ed02ba675]) successfully deleted
I20260925 04:55:57.988126 17125 ts_tablet_manager.cc:1918] T 5a6d1d7d0572439aafb46eec67d3366b P 898ebeecef974fbfa298d726ebd56bf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:55:57.988166 16615 catalog_manager.cc:4841] TS 0cc33a5ecb144bed985ef27de338db63 (127.6.29.130:46869): tablet 5a6d1d7d0572439aafb46eec67d3366b (table test-table [id=205a88f0a2684c7bb8cf1e8ed02ba675]) successfully deleted
I20260925 04:55:57.988351 17125 log.cc:1196] T 5a6d1d7d0572439aafb46eec67d3366b P 898ebeecef974fbfa298d726ebd56bf3: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790312053666503-6262-0/minicluster-data/ts-0/wal/wals/5a6d1d7d0572439aafb46eec67d3366b
I20260925 04:55:57.988956 17125 ts_tablet_manager.cc:1939] T 5a6d1d7d0572439aafb46eec67d3366b P 898ebeecef974fbfa298d726ebd56bf3: Deleting consensus metadata
I20260925 04:55:57.990932 16614 catalog_manager.cc:4841] TS 898ebeecef974fbfa298d726ebd56bf3 (127.6.29.129:34169): tablet 5a6d1d7d0572439aafb46eec67d3366b (table test-table [id=205a88f0a2684c7bb8cf1e8ed02ba675]) successfully deleted
Sep 25 04:55:57 dist-test-slave-xzjr krb5kdc[16583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.129@KRBTEST.COM
I20260925 04:55:58.003013 17100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:57.971518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53012 (local address 127.6.29.129:34169)
0925 04:55:57.971691 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:57.971696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:57.988860 (+ 17164us) server_negotiation.cc:407] Connection header received
0925 04:55:57.989083 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:57.989091 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:57.989180 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:57.989292 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:57.990534 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:57.991511 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:57.992881 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:57.993288 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:57.993429 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:57.997377 (+ 3948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:57.997389 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:57.997404 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:57.997438 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:58.000337 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:58.001001 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:58.001008 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:58.001011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:58.001141 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:58.001511 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:58.001520 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:58.001522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:58.001864 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:58.002185 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:58.002453 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 04:55:58.002658 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 04:55:58.003675 16812 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:55:58.095470 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 16689
I20260925 04:55:58.113746 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 16823
Sep 25 04:55:58 dist-test-slave-xzjr krb5kdc[16583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.130@KRBTEST.COM
I20260925 04:55:58.129256 17102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:58.114628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37884 (local address 127.6.29.130:46869)
0925 04:55:58.114814 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:58.114820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:58.114840 (+ 20us) server_negotiation.cc:407] Connection header received
0925 04:55:58.114972 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:58.114978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:58.115048 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:58.115168 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:58.116453 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:58.117181 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:58.118448 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:58.118848 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:58.118984 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:58.123281 (+ 4297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:58.123297 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:58.123313 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:58.123348 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:58.126386 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:58.127118 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:58.127125 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:58.127127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:58.127254 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:58.127627 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:58.127633 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:58.127636 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:58.128064 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:58.128370 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:58.128663 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 04:55:58.128879 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 04:55:58.129992 16946 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:55:58.224050 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 16823
I20260925 04:55:58.245965 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 16957
Sep 25 04:55:58 dist-test-slave-xzjr krb5kdc[16583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.131@KRBTEST.COM
I20260925 04:55:58.270563 17098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:55:58.246916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53186 (local address 127.6.29.131:39243)
0925 04:55:58.247168 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 04:55:58.247175 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:55:58.247203 (+ 28us) server_negotiation.cc:407] Connection header received
0925 04:55:58.247449 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:55:58.247456 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:55:58.247586 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:55:58.247742 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:55:58.249562 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:58.250981 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:58.252932 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:55:58.253761 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:55:58.253930 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:55:58.260019 (+ 6089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:55:58.260041 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:55:58.260063 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:55:58.260131 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:55:58.265915 (+ 5784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:58.267082 (+ 1167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:58.267091 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:58.267095 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:58.267497 (+ 402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:55:58.268078 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:55:58.268083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:55:58.268087 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:55:58.268668 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:55:58.268991 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0925 04:55:58.270088 (+ 1097us) server_negotiation.cc:299] Negotiation successful
0925 04:55:58.270263 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260925 04:55:58.271270 17080 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:55:58.403301 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 16957
I20260925 04:55:58.424489 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 16596
I20260925 04:55:58.425799 16658 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:55:58.534668 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 16596
2026-09-25T04:55:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4541 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:55:58 dist-test-slave-xzjr krb5kdc[17135](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:55:58 dist-test-slave-xzjr krb5kdc[17135](info): set up 2 sockets
Sep 25 04:55:58 dist-test-slave-xzjr krb5kdc[17135](info): commencing operation
krb5kdc: starting...
W20260925 04:55:59.645171 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 04:55:59 dist-test-slave-xzjr krb5kdc[17135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:55:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:55:59Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.29.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.29.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.29.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/krb5kdc/rangeradmin_127.6.29.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.29.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/krb5kdc/rangeradmin_127.6.29.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.29.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.29.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.29.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/krb5kdc/rangerlookup_127.6.29.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.29.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/krb5kdc/rangerlookup_127.6.29.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.29.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.29.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.29.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/krb5kdc/HTTP_127.6.29.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.29.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/krb5kdc/HTTP_127.6.29.148@KRBTEST.COM.keytab.
I20260925 04:55:59.979044 6262 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskjC0Yza/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/postgres -l logfile start
2026-09-25 04:56:02.631 UTC [17165] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 04:56:02.631 UTC [17165] LOG: listening on IPv4 address "127.6.29.148", port 45467
2026-09-25 04:56:02.635 UTC [17165] LOG: listening on Unix socket "/tmp/.s.PGSQL.45467"
2026-09-25 04:56:02.660 UTC [17168] LOG: database system was shut down at 2026-09-25 04:56:01 UTC
2026-09-25 04:56:02.664 UTC [17165] LOG: database system is ready to accept connections
I20260925 04:56:03.636979 6262 mini_postgres.cc:96] Postgres bound to 45467
2026-09-25 04:56:03.655 UTC [17176] FATAL: database "slave" does not exist
127.6.29.148:45467 - accepting connections
I20260925 04:56:03.656279 6262 mini_ranger.cc:161] Starting Ranger...
I20260925 04:56:03.683365 6262 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 04:56:04.198097 6262 mini_ranger.cc:87] Created ranger Postgres database
I20260925 04:56:04.198288 6262 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ranger-admin
2026-09-25 04:56:04,499 [I] DB FLAVOR :POSTGRES
2026-09-25 04:56:04,499 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 04:56:04,499 [I] Checking connection..
2026-09-25 04:56:04,500 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:45467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 04:56:04,763 [I] Checking connection passed.
2026-09-25 04:56:04,764 [I] --------- Verifying version history table ---------
2026-09-25 04:56:04,764 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:45467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 04:56:05,018 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 04:56:05,019 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:45467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 04:56:05,275 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 04:56:05,275 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 04:56:05,275 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:45467/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 04:56:05.517 UTC [17240] WARNING: there is no transaction in progress
2026-09-25 04:56:05,524 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 04:56:05,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:45467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 04:56:05,778 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 04:56:05,778 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 04:56:05,778 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:45467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 04:56:06,035 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:45467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 04:56:06,297 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:45467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-xzjr.c.gcp-upstream.internal','N') ;"
2026-09-25 04:56:06,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:45467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 04:56:06,812 [I] Table x_portal_user does not exist in database ranger
2026-09-25 04:56:06,813 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:45467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 04:56:07,048 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 04:56:07,048 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:45467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 04:56:07,297 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:45467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 04:56:07,565 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 04:56:07,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:45467/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 04:56:08.617 UTC [17348] WARNING: there is no transaction in progress
2026-09-25 04:56:08.635 UTC [17348] WARNING: there is no transaction in progress
2026-09-25 04:56:08.655 UTC [17348] WARNING: there is no transaction in progress
2026-09-25 04:56:08.672 UTC [17348] WARNING: there is no transaction in progress
2026-09-25 04:56:08.690 UTC [17348] WARNING: there is no transaction in progress
2026-09-25 04:56:08.709 UTC [17348] WARNING: there is no transaction in progress
2026-09-25 04:56:08.767 UTC [17348] WARNING: there is no transaction in progress
2026-09-25 04:56:08.775 UTC [17348] WARNING: there is no transaction in progress
2026-09-25 04:56:08.790 UTC [17348] WARNING: there is no transaction in progress
2026-09-25 04:56:08.801 UTC [17348] WARNING: there is no transaction in progress
2026-09-25 04:56:08.812 UTC [17348] WARNING: there is no transaction in progress
2026-09-25 04:56:08.828 UTC [17348] WARNING: there is no transaction in progress
2026-09-25 04:56:08.838 UTC [17348] WARNING: there is no transaction in progress
2026-09-25 04:56:08.845 UTC [17348] WARNING: there is no transaction in progress
2026-09-25 04:56:08.852 UTC [17348] WARNING: there is no transaction in progress
2026-09-25 04:56:08.859 UTC [17348] WARNING: there is no transaction in progress
2026-09-25 04:56:09,541 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 04:56:09,541 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:45467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 04:56:09,776 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 04:56:09,777 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:45467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 04:56:10,031 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 04:56:10,031 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:45467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 04:56:10,301 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 04:56:10,302 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:45467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 04:56:10,555 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:45467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 04:56:10,816 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:45467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-xzjr.c.gcp-upstream.internal';"
2026-09-25 04:56:11,073 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 04:56:11,073 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:45467/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 04:56:11,306 [I] DB_PATCHES have already been applied
I20260925 04:56:11.312836 6262 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ranger-admin:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskjC0Yza/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 04:56:11.312958 6262 mini_ranger.cc:193] Using host: 127.6.29.148
I20260925 04:56:11.329211 6262 mini_ranger.cc:227] Ranger admin URL: http://127.6.29.148:34499
Sep 25, 2026 4:56:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 4:56:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 4:56:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 4:56:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 4:56:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 4:56:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 4:56:12 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 04:56:12.495625 6262 mini_ranger.cc:160] Time spent starting Ranger: real 8.839s user 0.007s sys 0.033s
Sep 25, 2026 4:56:33 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 4:56:33 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 25, 2026 4:56:33 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 4:56:33 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 4:56:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 4:56:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 4:56:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 4:56:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 4:56:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 4:56:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 4:56:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 4:56:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 4:56:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 4:56:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 4:56:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 4:56:33 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 4:56:33 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 4:56:34 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 04:56:34.981129 6262 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.29.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.29.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.29.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/krb5kdc/rangerkms_127.6.29.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.29.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/krb5kdc/rangerkms_127.6.29.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.29.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.29.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.29.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/krb5kdc/HTTP_127.6.29.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.29.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/krb5kdc/HTTP_127.6.29.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 04:56:35 dist-test-slave-xzjr krb5kdc[17135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312195, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 04:56:35.147535 6262 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 04:56:35.169238 6262 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 04:56:36.106462 6262 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 04:56:36.106602 6262 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ranger-kms
I20260925 04:56:36.106762 6262 mini_ranger_kms.cc:227] Using postgres at 127.6.29.148:45467
2026-09-25 04:56:36,554 [I] DB FLAVOR :POSTGRES
2026-09-25 04:56:36,555 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 04:56:36,555 [I] Checking connection
2026-09-25 04:56:36,555 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:45467/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 04:56:36,818 [I] connection success
2026-09-25 04:56:36,818 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 04:56:36,818 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 04:56:36,818 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:45467/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 04:56:37,084 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 04:56:37,084 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 04:56:37,084 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 04:56:37,085 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:45467/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 04:56:37,350 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 04:56:37.850508 6262 mini_ranger_kms.cc:310] Created kms service
I20260925 04:56:38.019790 6262 mini_ranger_kms.cc:326] Created kudu user
I20260925 04:56:38.065955 6262 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 04:56:38 dist-test-slave-xzjr krb5kdc[17135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312195, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.29.148@KRBTEST.COM
I20260925 04:56:38.556843 6262 mini_ranger_kms.cc:384] Added ranger policy
I20260925 04:56:38.557159 6262 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ranger-kms:/tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 04:56:38.557227 6262 mini_ranger_kms.cc:240] Using host: 127.6.29.148
I20260925 04:56:38.586874 6262 mini_ranger_kms.cc:276] Ranger KMS PID: 17529
I20260925 04:56:38.587105 6262 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.29.148:57753
Sep 25, 2026 4:56:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 4:56:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 4:56:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 4:56:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 4:56:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 4:56:40 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 04:56:40.901402 6262 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.754s user 0.137s sys 0.337s
I20260925 04:56:40.901665 6262 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 04:56:40.901870 6262 mini_ranger_kms.cc:401] 127.6.29.148:57753/kms/v1/keys
Sep 25 04:56:40 dist-test-slave-xzjr krb5kdc[17135](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790312200, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.29.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 04:56:41 dist-test-slave-xzjr krb5kdc[17135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312200, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.29.148@KRBTEST.COM for HTTP/127.6.29.148@KRBTEST.COM
Sep 25 04:56:41 dist-test-slave-xzjr krb5kdc[17135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312200, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.29.148@KRBTEST.COM for HTTP/127.6.29.148@KRBTEST.COM
Sep 25 04:56:43 dist-test-slave-xzjr krb5kdc[17135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.29.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
I20260925 04:56:44.003846 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:36719
--webserver_interface=127.6.29.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:33141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:36719
--ranger_config_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.29.148:57753/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:56:44.345594 17579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:56:44.345896 17579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:56:44.345985 17579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:56:44.353228 17579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:56:44.353346 17579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:56:44.353392 17579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:56:44.353441 17579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:56:44.353482 17579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:56:44.363922 17579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:33141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.29.148:57753/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:36719
--ranger_config_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:36719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:56:44.365198 17579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:56:44.366477 17579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:56:44.378912 17584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:56:44.378988 17585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:56:44.378950 17587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:56:44.379410 17579 server_base.cc:1034] running on GCE node
I20260925 04:56:44.380473 17579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:56:44.381932 17579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:56:44.383134 17579 hybrid_clock.cc:648] HybridClock initialized: now 1790312204383092 us; error 40 us; skew 500 ppm
I20260925 04:56:44.383520 17579 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:56:44 dist-test-slave-xzjr krb5kdc[17135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312204, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:56:44.390009 17579 init.cc:377] Logged in from keytab as kudu/127.6.29.190@KRBTEST.COM (short username kudu)
I20260925 04:56:44.391615 17579 webserver.cc:460] Webserver started at http://127.6.29.190:35103/ using document root <none> and password file <none>
I20260925 04:56:44.392344 17579 fs_manager.cc:359] Metadata directory not provided
I20260925 04:56:44.392475 17579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:56:44.392810 17579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 04:56:44 dist-test-slave-xzjr krb5kdc[17135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312204, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for HTTP/127.6.29.148@KRBTEST.COM
I20260925 04:56:44.461488 17579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "9d770fbff972447faf59483b053a4466"
format_stamp: "Formatted at 2026-09-25 04:56:44 on dist-test-slave-xzjr"
server_key: "f8aafa767b43877b107e5773edf8b4e1"
server_key_iv: "54a6f713bd151087e408c03591f5012b"
server_key_version: "kuduclusterkey@0"
I20260925 04:56:44.462406 17579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "9d770fbff972447faf59483b053a4466"
format_stamp: "Formatted at 2026-09-25 04:56:44 on dist-test-slave-xzjr"
server_key: "f8aafa767b43877b107e5773edf8b4e1"
server_key_iv: "54a6f713bd151087e408c03591f5012b"
server_key_version: "kuduclusterkey@0"
I20260925 04:56:44.468353 17579 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260925 04:56:44.488349 17594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:56:44.490010 17579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 04:56:44.490231 17579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "9d770fbff972447faf59483b053a4466"
format_stamp: "Formatted at 2026-09-25 04:56:44 on dist-test-slave-xzjr"
server_key: "f8aafa767b43877b107e5773edf8b4e1"
server_key_iv: "54a6f713bd151087e408c03591f5012b"
server_key_version: "kuduclusterkey@0"
I20260925 04:56:44.490497 17579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:56:44.506091 17579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:56:44.510532 17579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:56:44.510871 17579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:56:44.531131 17579 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:36719
I20260925 04:56:44.531167 17645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:36719 every 8 connection(s)
I20260925 04:56:44.532793 17579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:56:44.537042 17646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:56:44.541831 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 17579
I20260925 04:56:44.542049 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 04:56:44.542846 6262 external_mini_cluster.cc:1428] Setting key d280d05c5169ad513a547d59c7d29ecb
I20260925 04:56:44.556404 17646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d770fbff972447faf59483b053a4466: Bootstrap starting.
I20260925 04:56:44.561664 17646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d770fbff972447faf59483b053a4466: Neither blocks nor log segments found. Creating new log.
I20260925 04:56:44.563277 17646 log.cc:824] T 00000000000000000000000000000000 P 9d770fbff972447faf59483b053a4466: Log is configured to *not* fsync() on all Append() calls
I20260925 04:56:44.568106 17646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d770fbff972447faf59483b053a4466: No bootstrap required, opened a new log
Sep 25 04:56:44 dist-test-slave-xzjr krb5kdc[17135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:56:44.579550 17646 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d770fbff972447faf59483b053a4466 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d770fbff972447faf59483b053a4466" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 36719 } }
I20260925 04:56:44.580152 17646 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d770fbff972447faf59483b053a4466 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:56:44.580294 17646 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d770fbff972447faf59483b053a4466 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d770fbff972447faf59483b053a4466, State: Initialized, Role: FOLLOWER
I20260925 04:56:44.581102 17646 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d770fbff972447faf59483b053a4466 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d770fbff972447faf59483b053a4466" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 36719 } }
I20260925 04:56:44.581390 17646 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d770fbff972447faf59483b053a4466 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:56:44.581557 17646 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d770fbff972447faf59483b053a4466 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:56:44.581740 17646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d770fbff972447faf59483b053a4466 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:56:44.583370 17649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:56:44.545219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55192 (local address 127.6.29.190:36719)
0925 04:56:44.556418 (+ 11199us) server_negotiation.cc:206] Beginning negotiation
0925 04:56:44.556437 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 04:56:44.560326 (+ 3889us) server_negotiation.cc:407] Connection header received
0925 04:56:44.562994 (+ 2668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:56:44.563029 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:56:44.563636 (+ 607us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:56:44.564233 (+ 597us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:56:44.565807 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:56:44.567172 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:56:44.568644 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:56:44.569349 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:56:44.569574 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:56:44.573617 (+ 4043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:56:44.573626 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:56:44.573647 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:56:44.573689 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:56:44.576942 (+ 3253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:56:44.577779 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:56:44.577786 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:56:44.577793 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:56:44.577962 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:56:44.578394 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:56:44.578406 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:56:44.578408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:56:44.578989 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:56:44.579330 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0925 04:56:44.579826 (+ 496us) server_negotiation.cc:299] Negotiation successful
0925 04:56:44.580490 (+ 664us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10921,"thread_start_us":10665,"threads_started":1}
I20260925 04:56:44.585393 17646 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d770fbff972447faf59483b053a4466 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d770fbff972447faf59483b053a4466" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 36719 } }
I20260925 04:56:44.586050 17646 leader_election.cc:302] T 00000000000000000000000000000000 P 9d770fbff972447faf59483b053a4466 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d770fbff972447faf59483b053a4466; no voters:
I20260925 04:56:44.586907 17646 leader_election.cc:288] T 00000000000000000000000000000000 P 9d770fbff972447faf59483b053a4466 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:56:44.586977 17651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d770fbff972447faf59483b053a4466 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:56:44.589212 17646 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d770fbff972447faf59483b053a4466 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:56:44.594501 17651 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d770fbff972447faf59483b053a4466 [term 1 LEADER]: Becoming Leader. State: Replica: 9d770fbff972447faf59483b053a4466, State: Running, Role: LEADER
I20260925 04:56:44.595397 17651 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d770fbff972447faf59483b053a4466 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d770fbff972447faf59483b053a4466" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 36719 } }
I20260925 04:56:44.606312 17652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d770fbff972447faf59483b053a4466 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d770fbff972447faf59483b053a4466" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d770fbff972447faf59483b053a4466" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 36719 } } }
I20260925 04:56:44.606917 17652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d770fbff972447faf59483b053a4466 [sys.catalog]: This master's current role is: LEADER
I20260925 04:56:44.615986 17653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d770fbff972447faf59483b053a4466 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d770fbff972447faf59483b053a4466. Latest consensus state: current_term: 1 leader_uuid: "9d770fbff972447faf59483b053a4466" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d770fbff972447faf59483b053a4466" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 36719 } } }
I20260925 04:56:44.616416 17653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d770fbff972447faf59483b053a4466 [sys.catalog]: This master's current role is: LEADER
I20260925 04:56:44.617964 17661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:56:44.624449 17661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:56:44.645341 17646 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 04:56:44.657214 17661 catalog_manager.cc:1347] Generated new cluster ID: 60caf6815f3d4afb9b70eb8006620183
I20260925 04:56:44.657428 17661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:56:44.681095 17661 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:56:44.683689 17661 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:56:44.695477 17661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d770fbff972447faf59483b053a4466: Generated new TSK 0
I20260925 04:56:44.696669 17661 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.29.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab.
Entry for principal kudu/127.6.29.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab.
I20260925 04:56:45.460902 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.29.148:57753/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.129:0
--local_ip_for_outbound_sockets=127.6.29.129
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:36719
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:33141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 04:56:45 dist-test-slave-xzjr krb5kdc[17135](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790312205, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 04:56:45.938768 17692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:56:45.939206 17692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:56:45.939429 17692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:56:45.951460 17692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:56:45.951625 17692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:56:45.951893 17692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.129
I20260925 04:56:45.968274 17692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:33141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.29.148:57753/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:36719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.129
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:56:45.970093 17692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:56:45.971967 17692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:56:45.996531 17702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:56:45.997170 17703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:56:46.002952 17705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:56:46.003530 17692 server_base.cc:1034] running on GCE node
I20260925 04:56:46.004437 17692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:56:46.005717 17692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:56:46.007036 17692 hybrid_clock.cc:648] HybridClock initialized: now 1790312206006960 us; error 61 us; skew 500 ppm
I20260925 04:56:46.007488 17692 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:56:46 dist-test-slave-xzjr krb5kdc[17135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312206, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:56:46.012590 17692 init.cc:377] Logged in from keytab as kudu/127.6.29.129@KRBTEST.COM (short username kudu)
I20260925 04:56:46.020947 17692 webserver.cc:460] Webserver started at http://127.6.29.129:42355/ using document root <none> and password file <none>
I20260925 04:56:46.021780 17692 fs_manager.cc:359] Metadata directory not provided
I20260925 04:56:46.021922 17692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:56:46.022257 17692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 04:56:46 dist-test-slave-xzjr krb5kdc[17135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312206, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for HTTP/127.6.29.148@KRBTEST.COM
I20260925 04:56:46.039535 17692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-0/data/instance:
uuid: "72a50bf0c64f4315be657e2a9031e6a0"
format_stamp: "Formatted at 2026-09-25 04:56:46 on dist-test-slave-xzjr"
server_key: "27223b526f2d3fbc24f388b38b353e4e"
server_key_iv: "812fb4003571c2c4fe023ebadac573f1"
server_key_version: "kuduclusterkey@0"
I20260925 04:56:46.040572 17692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance:
uuid: "72a50bf0c64f4315be657e2a9031e6a0"
format_stamp: "Formatted at 2026-09-25 04:56:46 on dist-test-slave-xzjr"
server_key: "27223b526f2d3fbc24f388b38b353e4e"
server_key_iv: "812fb4003571c2c4fe023ebadac573f1"
server_key_version: "kuduclusterkey@0"
I20260925 04:56:46.046741 17692 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20260925 04:56:46.078792 17712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:56:46.080981 17692 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.005s sys 0.003s
I20260925 04:56:46.081274 17692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-0/wal
uuid: "72a50bf0c64f4315be657e2a9031e6a0"
format_stamp: "Formatted at 2026-09-25 04:56:46 on dist-test-slave-xzjr"
server_key: "27223b526f2d3fbc24f388b38b353e4e"
server_key_iv: "812fb4003571c2c4fe023ebadac573f1"
server_key_version: "kuduclusterkey@0"
I20260925 04:56:46.081641 17692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:56:46.133301 17692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:56:46.139128 17692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:56:46.139537 17692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:56:46.147132 17692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:56:46.152218 17692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:56:46.152336 17692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:56:46.152607 17692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:56:46.152751 17692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 04:56:46 dist-test-slave-xzjr krb5kdc[17135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312205, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for HTTP/127.6.29.148@KRBTEST.COM
I20260925 04:56:46.320443 17692 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.129:41513
I20260925 04:56:46.320498 17824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.129:41513 every 8 connection(s)
I20260925 04:56:46.322250 17692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
I20260925 04:56:46.323709 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 17692
I20260925 04:56:46.323946 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance
I20260925 04:56:46.324769 6262 external_mini_cluster.cc:1428] Setting key 0d081178450715960ed9a299a11f1464
Sep 25 04:56:46 dist-test-slave-xzjr krb5kdc[17135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312206, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:56:46.361546 17828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:56:46.326902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:54715 (local address 127.6.29.190:36719)
0925 04:56:46.337007 (+ 10105us) server_negotiation.cc:206] Beginning negotiation
0925 04:56:46.337014 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:56:46.338352 (+ 1338us) server_negotiation.cc:407] Connection header received
0925 04:56:46.341105 (+ 2753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:56:46.341112 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:56:46.341232 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:56:46.341379 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:56:46.343662 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:56:46.344497 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:56:46.347303 (+ 2806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:56:46.347728 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:56:46.347922 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:56:46.353381 (+ 5459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:56:46.353391 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:56:46.353399 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:56:46.353474 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:56:46.357938 (+ 4464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:56:46.358800 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:56:46.358806 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:56:46.358810 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:56:46.358986 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:56:46.359460 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:56:46.359468 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:56:46.359473 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:56:46.359945 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:56:46.360108 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 04:56:46.360888 (+ 780us) server_negotiation.cc:299] Negotiation successful
0925 04:56:46.361148 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9941,"thread_start_us":9409,"threads_started":1}
I20260925 04:56:46.362730 17827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:56:46.326178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:36719 (local address 127.6.29.129:54715)
0925 04:56:46.337808 (+ 11630us) negotiation.cc:107] Waiting for socket to connect
0925 04:56:46.337912 (+ 104us) client_negotiation.cc:174] Beginning negotiation
0925 04:56:46.340716 (+ 2804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:56:46.341723 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:56:46.341739 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:56:46.342450 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:56:46.343442 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:56:46.343476 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:56:46.344669 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:56:46.344677 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:56:46.347042 (+ 2365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:56:46.347062 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:56:46.348263 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:56:46.348273 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:56:46.348667 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:56:46.349851 (+ 1184us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:56:46.349888 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:56:46.353040 (+ 3152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:56:46.358133 (+ 5093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:56:46.358151 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:56:46.358167 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:56:46.358652 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:56:46.359117 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:56:46.359121 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:56:46.359123 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:56:46.359333 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:56:46.360103 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:56:46.360112 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:56:46.360507 (+ 395us) client_negotiation.cc:769] Sending connection context
0925 04:56:46.360805 (+ 298us) client_negotiation.cc:240] Negotiation successful
0925 04:56:46.361413 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11138,"thread_start_us":10887,"threads_started":1}
I20260925 04:56:46.364464 17825 heartbeater.cc:347] Connected to a master server at 127.6.29.190:36719
WARNING: no policy specified for kudu/127.6.29.130@KRBTEST.COM; defaulting to no policy
I20260925 04:56:46.364791 17825 heartbeater.cc:464] Registering TS with master...
I20260925 04:56:46.365657 17825 heartbeater.cc:511] Master 127.6.29.190:36719 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.130@KRBTEST.COM" created.
I20260925 04:56:46.369079 17611 ts_manager.cc:194] Registered new tserver with Master: 72a50bf0c64f4315be657e2a9031e6a0 (127.6.29.129:41513)
I20260925 04:56:46.371335 17611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.129@KRBTEST.COM'} at 127.6.29.129:54715
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab.
Entry for principal kudu/127.6.29.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab.
I20260925 04:56:46.403041 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.29.148:57753/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.130:0
--local_ip_for_outbound_sockets=127.6.29.130
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:36719
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:33141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 04:56:46 dist-test-slave-xzjr krb5kdc[17135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312205, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for HTTP/127.6.29.148@KRBTEST.COM
W20260925 04:56:46.874816 17831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:56:46.875169 17831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:56:46.875339 17831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:56:46.883131 17831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:56:46.883313 17831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:56:46.883499 17831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.130
I20260925 04:56:46.893795 17831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:33141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.29.148:57753/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:36719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.130
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:56:46.895257 17831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:56:46.896642 17831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:56:46.910598 17848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:56:46.910670 17846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:56:46.910637 17845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:56:46.911592 17831 server_base.cc:1034] running on GCE node
I20260925 04:56:46.912242 17831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:56:46.913208 17831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:56:46.914407 17831 hybrid_clock.cc:648] HybridClock initialized: now 1790312206914345 us; error 87 us; skew 500 ppm
I20260925 04:56:46.914693 17831 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:56:46 dist-test-slave-xzjr krb5kdc[17135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312206, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:56:46.917743 17831 init.cc:377] Logged in from keytab as kudu/127.6.29.130@KRBTEST.COM (short username kudu)
I20260925 04:56:46.919234 17831 webserver.cc:460] Webserver started at http://127.6.29.130:37129/ using document root <none> and password file <none>
I20260925 04:56:46.919993 17831 fs_manager.cc:359] Metadata directory not provided
I20260925 04:56:46.920137 17831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:56:46.920469 17831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 04:56:46 dist-test-slave-xzjr krb5kdc[17135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312206, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for HTTP/127.6.29.148@KRBTEST.COM
I20260925 04:56:46.937338 17831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-1/data/instance:
uuid: "2c2132a6049c4ca1b1b667a07f80a007"
format_stamp: "Formatted at 2026-09-25 04:56:46 on dist-test-slave-xzjr"
server_key: "cefbc2445ef42dd27c150f5e2e5ad105"
server_key_iv: "06a6346634b108a2a2374dc528915c83"
server_key_version: "kuduclusterkey@0"
I20260925 04:56:46.938167 17831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance:
uuid: "2c2132a6049c4ca1b1b667a07f80a007"
format_stamp: "Formatted at 2026-09-25 04:56:46 on dist-test-slave-xzjr"
server_key: "cefbc2445ef42dd27c150f5e2e5ad105"
server_key_iv: "06a6346634b108a2a2374dc528915c83"
server_key_version: "kuduclusterkey@0"
I20260925 04:56:46.943895 17831 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260925 04:56:46.960268 17855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:56:46.961886 17831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 04:56:46.962102 17831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-1/wal
uuid: "2c2132a6049c4ca1b1b667a07f80a007"
format_stamp: "Formatted at 2026-09-25 04:56:46 on dist-test-slave-xzjr"
server_key: "cefbc2445ef42dd27c150f5e2e5ad105"
server_key_iv: "06a6346634b108a2a2374dc528915c83"
server_key_version: "kuduclusterkey@0"
I20260925 04:56:46.962352 17831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:56:46.983559 17831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:56:46.989121 17831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:56:46.989431 17831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:56:46.990540 17831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:56:46.991976 17831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:56:46.992095 17831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:56:46.992251 17831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:56:46.992324 17831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:56:47.030126 17831 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.130:44493
I20260925 04:56:47.030175 17967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.130:44493 every 8 connection(s)
I20260925 04:56:47.031489 17831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
I20260925 04:56:47.036981 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 17831
I20260925 04:56:47.037182 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance
I20260925 04:56:47.037614 6262 external_mini_cluster.cc:1428] Setting key e4d1e86e74de07f8563f25740470fb2f
Sep 25 04:56:47 dist-test-slave-xzjr krb5kdc[17135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312206, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:56:47.054695 17970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:56:47.034628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:46149 (local address 127.6.29.190:36719)
0925 04:56:47.035124 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0925 04:56:47.035129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:56:47.036028 (+ 899us) server_negotiation.cc:407] Connection header received
0925 04:56:47.038147 (+ 2119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:56:47.038153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:56:47.038256 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:56:47.038395 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:56:47.040341 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:56:47.041090 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:56:47.043138 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:56:47.043714 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:56:47.043903 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:56:47.047634 (+ 3731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:56:47.047644 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:56:47.047652 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:56:47.047700 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:56:47.050551 (+ 2851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:56:47.051624 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:56:47.051627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:56:47.051630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:56:47.051749 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:56:47.052382 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:56:47.052385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:56:47.052387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:56:47.052687 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:56:47.052800 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 04:56:47.053803 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0925 04:56:47.054415 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":282,"threads_started":1}
I20260925 04:56:47.056015 17971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:56:47.035049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:36719 (local address 127.6.29.130:46149)
0925 04:56:47.035855 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0925 04:56:47.035897 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 04:56:47.037826 (+ 1929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:56:47.038726 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:56:47.038740 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:56:47.039368 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:56:47.040159 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:56:47.040181 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:56:47.041275 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:56:47.041279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:56:47.042961 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:56:47.042990 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:56:47.044010 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:56:47.044019 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:56:47.044298 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:56:47.045044 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:56:47.045076 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:56:47.047366 (+ 2290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:56:47.050736 (+ 3370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:56:47.050750 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:56:47.050773 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:56:47.051491 (+ 718us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:56:47.051906 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:56:47.051914 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:56:47.051921 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:56:47.052269 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:56:47.052836 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:56:47.052848 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:56:47.053395 (+ 547us) client_negotiation.cc:769] Sending connection context
0925 04:56:47.053733 (+ 338us) client_negotiation.cc:240] Negotiation successful
0925 04:56:47.054389 (+ 656us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":309,"threads_started":1}
I20260925 04:56:47.057699 17968 heartbeater.cc:347] Connected to a master server at 127.6.29.190:36719
I20260925 04:56:47.058036 17968 heartbeater.cc:464] Registering TS with master...
I20260925 04:56:47.058815 17968 heartbeater.cc:511] Master 127.6.29.190:36719 requested a full tablet report, sending...
I20260925 04:56:47.060437 17611 ts_manager.cc:194] Registered new tserver with Master: 2c2132a6049c4ca1b1b667a07f80a007 (127.6.29.130:44493)
I20260925 04:56:47.061673 17611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.130@KRBTEST.COM'} at 127.6.29.130:46149
WARNING: no policy specified for kudu/127.6.29.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab.
Entry for principal kudu/127.6.29.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab.
I20260925 04:56:47.095257 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.29.148:57753/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.131:0
--local_ip_for_outbound_sockets=127.6.29.131
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:36719
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:33141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:56:47.375564 17825 heartbeater.cc:503] Master 127.6.29.190:36719 was elected leader, sending a full tablet report...
W20260925 04:56:47.443045 17974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:56:47.443329 17974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:56:47.443500 17974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:56:47.450820 17974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:56:47.450929 17974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:56:47.451135 17974 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.131
I20260925 04:56:47.461325 17974 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:33141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.6.29.148:57753/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:36719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.131
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:56:47.462715 17974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:56:47.464118 17974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:56:47.478452 17980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:56:47.478721 17982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:56:47.478564 17979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:56:47.479501 17974 server_base.cc:1034] running on GCE node
I20260925 04:56:47.480260 17974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:56:47.481240 17974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:56:47.482524 17974 hybrid_clock.cc:648] HybridClock initialized: now 1790312207482442 us; error 58 us; skew 500 ppm
I20260925 04:56:47.482957 17974 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:56:47 dist-test-slave-xzjr krb5kdc[17135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312207, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:56:47.486369 17974 init.cc:377] Logged in from keytab as kudu/127.6.29.131@KRBTEST.COM (short username kudu)
I20260925 04:56:47.488029 17974 webserver.cc:460] Webserver started at http://127.6.29.131:41253/ using document root <none> and password file <none>
I20260925 04:56:47.488823 17974 fs_manager.cc:359] Metadata directory not provided
I20260925 04:56:47.488961 17974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:56:47.489362 17974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 04:56:47 dist-test-slave-xzjr krb5kdc[17135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312207, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for HTTP/127.6.29.148@KRBTEST.COM
I20260925 04:56:47.506973 17974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-2/data/instance:
uuid: "8fea8b6b7a1a4b01acbb4a2bb73d7541"
format_stamp: "Formatted at 2026-09-25 04:56:47 on dist-test-slave-xzjr"
server_key: "7810b02fd4c92a7e12bc01a23e08af88"
server_key_iv: "9e4814d633efa646e49fa559ebfb6051"
server_key_version: "kuduclusterkey@0"
I20260925 04:56:47.508145 17974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance:
uuid: "8fea8b6b7a1a4b01acbb4a2bb73d7541"
format_stamp: "Formatted at 2026-09-25 04:56:47 on dist-test-slave-xzjr"
server_key: "7810b02fd4c92a7e12bc01a23e08af88"
server_key_iv: "9e4814d633efa646e49fa559ebfb6051"
server_key_version: "kuduclusterkey@0"
I20260925 04:56:47.513929 17974 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 04:56:47.531423 17989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:56:47.533058 17974 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 04:56:47.533243 17974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-2/wal
uuid: "8fea8b6b7a1a4b01acbb4a2bb73d7541"
format_stamp: "Formatted at 2026-09-25 04:56:47 on dist-test-slave-xzjr"
server_key: "7810b02fd4c92a7e12bc01a23e08af88"
server_key_iv: "9e4814d633efa646e49fa559ebfb6051"
server_key_version: "kuduclusterkey@0"
I20260925 04:56:47.533497 17974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:56:47.549185 17974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:56:47.553452 17974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:56:47.553762 17974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:56:47.554803 17974 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:56:47.556526 17974 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:56:47.556629 17974 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:56:47.556770 17974 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:56:47.556847 17974 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:56:47.595580 17974 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.131:44627
I20260925 04:56:47.595638 18101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.131:44627 every 8 connection(s)
I20260925 04:56:47.596961 17974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
I20260925 04:56:47.599538 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 17974
I20260925 04:56:47.599721 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance
I20260925 04:56:47.600340 6262 external_mini_cluster.cc:1428] Setting key 523a9a05fee3005438962b88142285a2
Sep 25 04:56:47 dist-test-slave-xzjr krb5kdc[17135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312207, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:56:47.617977 18104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:56:47.599738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:53881 (local address 127.6.29.190:36719)
0925 04:56:47.600782 (+ 1044us) server_negotiation.cc:206] Beginning negotiation
0925 04:56:47.600787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:56:47.601859 (+ 1072us) server_negotiation.cc:407] Connection header received
0925 04:56:47.603890 (+ 2031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:56:47.603895 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:56:47.604004 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:56:47.604126 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:56:47.605567 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:56:47.606276 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:56:47.607851 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:56:47.608322 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:56:47.608463 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:56:47.611944 (+ 3481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:56:47.611949 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:56:47.611953 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:56:47.611985 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:56:47.614649 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:56:47.615516 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:56:47.615522 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:56:47.615525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:56:47.615671 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:56:47.616165 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:56:47.616169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:56:47.616171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:56:47.616445 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:56:47.616598 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 04:56:47.617386 (+ 788us) server_negotiation.cc:299] Negotiation successful
0925 04:56:47.617616 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":938,"thread_start_us":758,"threads_started":1}
I20260925 04:56:47.619055 18105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:56:47.600465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:36719 (local address 127.6.29.131:53881)
0925 04:56:47.601698 (+ 1233us) negotiation.cc:107] Waiting for socket to connect
0925 04:56:47.601760 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 04:56:47.603597 (+ 1837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:56:47.604368 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:56:47.604377 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:56:47.604819 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:56:47.605402 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:56:47.605418 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:56:47.606427 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:56:47.606437 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:56:47.607685 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:56:47.607696 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:56:47.608569 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:56:47.608576 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:56:47.608809 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:56:47.609535 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:56:47.609564 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:56:47.611641 (+ 2077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:56:47.614786 (+ 3145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:56:47.614797 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:56:47.614825 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:56:47.615386 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:56:47.615820 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:56:47.615831 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:56:47.615834 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:56:47.616035 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:56:47.616588 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:56:47.616596 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:56:47.616941 (+ 345us) client_negotiation.cc:769] Sending connection context
0925 04:56:47.617294 (+ 353us) client_negotiation.cc:240] Negotiation successful
0925 04:56:47.617853 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":448,"threads_started":1}
I20260925 04:56:47.620450 18102 heartbeater.cc:347] Connected to a master server at 127.6.29.190:36719
I20260925 04:56:47.620765 18102 heartbeater.cc:464] Registering TS with master...
I20260925 04:56:47.621495 18102 heartbeater.cc:511] Master 127.6.29.190:36719 requested a full tablet report, sending...
I20260925 04:56:47.622913 17611 ts_manager.cc:194] Registered new tserver with Master: 8fea8b6b7a1a4b01acbb4a2bb73d7541 (127.6.29.131:44627)
I20260925 04:56:47.623816 17611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.131@KRBTEST.COM'} at 127.6.29.131:53881
I20260925 04:56:47.628165 6262 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:56:47.645789 18104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:56:47.633099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55198 (local address 127.6.29.190:36719)
0925 04:56:47.633323 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 04:56:47.633333 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:56:47.633671 (+ 338us) server_negotiation.cc:407] Connection header received
0925 04:56:47.633894 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:56:47.633898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:56:47.633969 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:56:47.634078 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:56:47.635517 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:56:47.636297 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:56:47.637721 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:56:47.638151 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:56:47.638287 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:56:47.640787 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:56:47.640792 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:56:47.640795 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:56:47.640825 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:56:47.643081 (+ 2256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:56:47.643790 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:56:47.643796 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:56:47.643800 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:56:47.643928 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:56:47.644314 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:56:47.644320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:56:47.644323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:56:47.644601 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:56:47.644855 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0925 04:56:47.645211 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 04:56:47.645454 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 04:56:47.649993 17611 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55198:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:56:47.657696 17611 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:56:47.681551 18118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:56:47.671084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32854 (local address 127.6.29.131:44627)
0925 04:56:47.673399 (+ 2315us) server_negotiation.cc:206] Beginning negotiation
0925 04:56:47.673405 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:56:47.673428 (+ 23us) server_negotiation.cc:407] Connection header received
0925 04:56:47.673538 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:56:47.673545 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:56:47.673786 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:56:47.674001 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:56:47.674838 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:56:47.675893 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:56:47.678917 (+ 3024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:56:47.680386 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:56:47.680524 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:56:47.680757 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0925 04:56:47.681013 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 04:56:47.681167 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2195,"thread_start_us":2084,"threads_started":1}
I20260925 04:56:47.681551 18115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:56:47.670443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.131:44627 (local address 127.0.0.1:32854)
0925 04:56:47.672272 (+ 1829us) negotiation.cc:107] Waiting for socket to connect
0925 04:56:47.672319 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 04:56:47.672931 (+ 612us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:56:47.674034 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:56:47.674047 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:56:47.674093 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:56:47.674631 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:56:47.674647 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:56:47.676021 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:56:47.676029 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:56:47.678731 (+ 2702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:56:47.678744 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:56:47.680582 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:56:47.680589 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:56:47.680779 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:56:47.680828 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 04:56:47.680971 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 04:56:47.681157 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":993,"threads_started":1}
I20260925 04:56:47.685688 18037 tablet_service.cc:1467] Processing CreateTablet for tablet 56df987f276749098b0e06659c362a03 (DEFAULT_TABLE table=test-table [id=6809f77e0ac24ddaacefbad92ee2caf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:56:47.687171 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:56:47.670681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:41513 (local address 127.0.0.1:44680)
0925 04:56:47.677816 (+ 7135us) negotiation.cc:107] Waiting for socket to connect
0925 04:56:47.677835 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 04:56:47.678043 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:56:47.679041 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:56:47.679048 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:56:47.679080 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:56:47.679541 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:56:47.679554 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:56:47.682396 (+ 2842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:56:47.682401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:56:47.684478 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:56:47.684494 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:56:47.686349 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:56:47.686355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:56:47.686535 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:56:47.686577 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 04:56:47.686670 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 04:56:47.686807 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7011,"thread_start_us":6885,"threads_started":1}
I20260925 04:56:47.687332 18037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56df987f276749098b0e06659c362a03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:56:47.688114 18117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:56:47.671097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44680 (local address 127.6.29.129:41513)
0925 04:56:47.672878 (+ 1781us) server_negotiation.cc:206] Beginning negotiation
0925 04:56:47.672887 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:56:47.677940 (+ 5053us) server_negotiation.cc:407] Connection header received
0925 04:56:47.678269 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:56:47.678275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:56:47.678545 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:56:47.678719 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:56:47.679678 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:56:47.681236 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:56:47.684657 (+ 3421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:56:47.686158 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:56:47.686303 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:56:47.686540 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0925 04:56:47.687562 (+ 1022us) server_negotiation.cc:299] Negotiation successful
0925 04:56:47.687802 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1223,"thread_start_us":1092,"threads_started":1}
I20260925 04:56:47.689033 18119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:56:47.678178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42434 (local address 127.6.29.130:44493)
0925 04:56:47.678772 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0925 04:56:47.678783 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:56:47.679479 (+ 696us) server_negotiation.cc:407] Connection header received
0925 04:56:47.679655 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:56:47.679662 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:56:47.679916 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:56:47.680099 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:56:47.680675 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:56:47.682526 (+ 1851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:56:47.686144 (+ 3618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:56:47.687954 (+ 1810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:56:47.688109 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:56:47.688477 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0925 04:56:47.688553 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 04:56:47.688720 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":323,"threads_started":1}
I20260925 04:56:47.689081 18120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:56:47.678787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:44493 (local address 127.0.0.1:42434)
0925 04:56:47.679338 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0925 04:56:47.679363 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 04:56:47.679497 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:56:47.680103 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:56:47.680108 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:56:47.680133 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:56:47.680456 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:56:47.680466 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:56:47.683953 (+ 3487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:56:47.683957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:56:47.685895 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:56:47.685910 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:56:47.688195 (+ 2285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:56:47.688203 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:56:47.688373 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:56:47.688415 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 04:56:47.688503 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 04:56:47.688632 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"mutex_wait_us":39,"thread_start_us":274,"threads_started":1}
I20260925 04:56:47.691898 17760 tablet_service.cc:1467] Processing CreateTablet for tablet 56df987f276749098b0e06659c362a03 (DEFAULT_TABLE table=test-table [id=6809f77e0ac24ddaacefbad92ee2caf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:56:47.692852 17903 tablet_service.cc:1467] Processing CreateTablet for tablet 56df987f276749098b0e06659c362a03 (DEFAULT_TABLE table=test-table [id=6809f77e0ac24ddaacefbad92ee2caf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:56:47.693464 17760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56df987f276749098b0e06659c362a03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:56:47.694720 17903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56df987f276749098b0e06659c362a03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:56:47.703401 18121 tablet_bootstrap.cc:492] T 56df987f276749098b0e06659c362a03 P 8fea8b6b7a1a4b01acbb4a2bb73d7541: Bootstrap starting.
I20260925 04:56:47.705994 18122 tablet_bootstrap.cc:492] T 56df987f276749098b0e06659c362a03 P 72a50bf0c64f4315be657e2a9031e6a0: Bootstrap starting.
I20260925 04:56:47.707578 18121 tablet_bootstrap.cc:654] T 56df987f276749098b0e06659c362a03 P 8fea8b6b7a1a4b01acbb4a2bb73d7541: Neither blocks nor log segments found. Creating new log.
I20260925 04:56:47.708671 18121 log.cc:824] T 56df987f276749098b0e06659c362a03 P 8fea8b6b7a1a4b01acbb4a2bb73d7541: Log is configured to *not* fsync() on all Append() calls
I20260925 04:56:47.709968 18122 tablet_bootstrap.cc:654] T 56df987f276749098b0e06659c362a03 P 72a50bf0c64f4315be657e2a9031e6a0: Neither blocks nor log segments found. Creating new log.
I20260925 04:56:47.711683 18123 tablet_bootstrap.cc:492] T 56df987f276749098b0e06659c362a03 P 2c2132a6049c4ca1b1b667a07f80a007: Bootstrap starting.
I20260925 04:56:47.711941 18122 log.cc:824] T 56df987f276749098b0e06659c362a03 P 72a50bf0c64f4315be657e2a9031e6a0: Log is configured to *not* fsync() on all Append() calls
I20260925 04:56:47.716441 18123 tablet_bootstrap.cc:654] T 56df987f276749098b0e06659c362a03 P 2c2132a6049c4ca1b1b667a07f80a007: Neither blocks nor log segments found. Creating new log.
I20260925 04:56:47.717644 18123 log.cc:824] T 56df987f276749098b0e06659c362a03 P 2c2132a6049c4ca1b1b667a07f80a007: Log is configured to *not* fsync() on all Append() calls
I20260925 04:56:47.717846 18121 tablet_bootstrap.cc:492] T 56df987f276749098b0e06659c362a03 P 8fea8b6b7a1a4b01acbb4a2bb73d7541: No bootstrap required, opened a new log
I20260925 04:56:47.717846 18122 tablet_bootstrap.cc:492] T 56df987f276749098b0e06659c362a03 P 72a50bf0c64f4315be657e2a9031e6a0: No bootstrap required, opened a new log
I20260925 04:56:47.718156 18121 ts_tablet_manager.cc:1397] T 56df987f276749098b0e06659c362a03 P 8fea8b6b7a1a4b01acbb4a2bb73d7541: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20260925 04:56:47.718156 18122 ts_tablet_manager.cc:1397] T 56df987f276749098b0e06659c362a03 P 72a50bf0c64f4315be657e2a9031e6a0: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.006s
I20260925 04:56:47.720974 18123 tablet_bootstrap.cc:492] T 56df987f276749098b0e06659c362a03 P 2c2132a6049c4ca1b1b667a07f80a007: No bootstrap required, opened a new log
I20260925 04:56:47.721251 18123 ts_tablet_manager.cc:1397] T 56df987f276749098b0e06659c362a03 P 2c2132a6049c4ca1b1b667a07f80a007: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260925 04:56:47.728827 18123 raft_consensus.cc:348] T 56df987f276749098b0e06659c362a03 P 2c2132a6049c4ca1b1b667a07f80a007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fea8b6b7a1a4b01acbb4a2bb73d7541" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 44627 } } peers { permanent_uuid: "72a50bf0c64f4315be657e2a9031e6a0" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41513 } } peers { permanent_uuid: "2c2132a6049c4ca1b1b667a07f80a007" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44493 } }
I20260925 04:56:47.729241 18123 raft_consensus.cc:374] T 56df987f276749098b0e06659c362a03 P 2c2132a6049c4ca1b1b667a07f80a007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:56:47.729343 18123 raft_consensus.cc:729] T 56df987f276749098b0e06659c362a03 P 2c2132a6049c4ca1b1b667a07f80a007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c2132a6049c4ca1b1b667a07f80a007, State: Initialized, Role: FOLLOWER
I20260925 04:56:47.729823 18121 raft_consensus.cc:348] T 56df987f276749098b0e06659c362a03 P 8fea8b6b7a1a4b01acbb4a2bb73d7541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fea8b6b7a1a4b01acbb4a2bb73d7541" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 44627 } } peers { permanent_uuid: "72a50bf0c64f4315be657e2a9031e6a0" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41513 } } peers { permanent_uuid: "2c2132a6049c4ca1b1b667a07f80a007" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44493 } }
I20260925 04:56:47.729823 18122 raft_consensus.cc:348] T 56df987f276749098b0e06659c362a03 P 72a50bf0c64f4315be657e2a9031e6a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fea8b6b7a1a4b01acbb4a2bb73d7541" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 44627 } } peers { permanent_uuid: "72a50bf0c64f4315be657e2a9031e6a0" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41513 } } peers { permanent_uuid: "2c2132a6049c4ca1b1b667a07f80a007" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44493 } }
I20260925 04:56:47.730058 18123 consensus_queue.cc:260] T 56df987f276749098b0e06659c362a03 P 2c2132a6049c4ca1b1b667a07f80a007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fea8b6b7a1a4b01acbb4a2bb73d7541" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 44627 } } peers { permanent_uuid: "72a50bf0c64f4315be657e2a9031e6a0" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41513 } } peers { permanent_uuid: "2c2132a6049c4ca1b1b667a07f80a007" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44493 } }
I20260925 04:56:47.730275 18121 raft_consensus.cc:374] T 56df987f276749098b0e06659c362a03 P 8fea8b6b7a1a4b01acbb4a2bb73d7541 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:56:47.730332 18122 raft_consensus.cc:374] T 56df987f276749098b0e06659c362a03 P 72a50bf0c64f4315be657e2a9031e6a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:56:47.730382 18121 raft_consensus.cc:729] T 56df987f276749098b0e06659c362a03 P 8fea8b6b7a1a4b01acbb4a2bb73d7541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fea8b6b7a1a4b01acbb4a2bb73d7541, State: Initialized, Role: FOLLOWER
I20260925 04:56:47.730433 18122 raft_consensus.cc:729] T 56df987f276749098b0e06659c362a03 P 72a50bf0c64f4315be657e2a9031e6a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72a50bf0c64f4315be657e2a9031e6a0, State: Initialized, Role: FOLLOWER
I20260925 04:56:47.730998 18121 consensus_queue.cc:260] T 56df987f276749098b0e06659c362a03 P 8fea8b6b7a1a4b01acbb4a2bb73d7541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fea8b6b7a1a4b01acbb4a2bb73d7541" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 44627 } } peers { permanent_uuid: "72a50bf0c64f4315be657e2a9031e6a0" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41513 } } peers { permanent_uuid: "2c2132a6049c4ca1b1b667a07f80a007" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44493 } }
I20260925 04:56:47.731006 18122 consensus_queue.cc:260] T 56df987f276749098b0e06659c362a03 P 72a50bf0c64f4315be657e2a9031e6a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fea8b6b7a1a4b01acbb4a2bb73d7541" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 44627 } } peers { permanent_uuid: "72a50bf0c64f4315be657e2a9031e6a0" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41513 } } peers { permanent_uuid: "2c2132a6049c4ca1b1b667a07f80a007" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44493 } }
I20260925 04:56:47.731539 17968 heartbeater.cc:503] Master 127.6.29.190:36719 was elected leader, sending a full tablet report...
I20260925 04:56:47.732323 18123 ts_tablet_manager.cc:1428] T 56df987f276749098b0e06659c362a03 P 2c2132a6049c4ca1b1b667a07f80a007: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260925 04:56:47.734107 18121 ts_tablet_manager.cc:1428] T 56df987f276749098b0e06659c362a03 P 8fea8b6b7a1a4b01acbb4a2bb73d7541: Time spent starting tablet: real 0.016s user 0.010s sys 0.007s
I20260925 04:56:47.734385 18122 ts_tablet_manager.cc:1428] T 56df987f276749098b0e06659c362a03 P 72a50bf0c64f4315be657e2a9031e6a0: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260925 04:56:47.735086 18102 heartbeater.cc:503] Master 127.6.29.190:36719 was elected leader, sending a full tablet report...
W20260925 04:56:47.784192 17969 tablet.cc:2369] T 56df987f276749098b0e06659c362a03 P 2c2132a6049c4ca1b1b667a07f80a007: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:56:47.837920 17826 tablet.cc:2369] T 56df987f276749098b0e06659c362a03 P 72a50bf0c64f4315be657e2a9031e6a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:56:47.848845 18103 tablet.cc:2369] T 56df987f276749098b0e06659c362a03 P 8fea8b6b7a1a4b01acbb4a2bb73d7541: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:56:47.865511 18127 raft_consensus.cc:482] T 56df987f276749098b0e06659c362a03 P 2c2132a6049c4ca1b1b667a07f80a007 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:56:47.865854 18127 raft_consensus.cc:504] T 56df987f276749098b0e06659c362a03 P 2c2132a6049c4ca1b1b667a07f80a007 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fea8b6b7a1a4b01acbb4a2bb73d7541" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 44627 } } peers { permanent_uuid: "72a50bf0c64f4315be657e2a9031e6a0" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41513 } } peers { permanent_uuid: "2c2132a6049c4ca1b1b667a07f80a007" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44493 } }
I20260925 04:56:47.868260 18127 leader_election.cc:288] T 56df987f276749098b0e06659c362a03 P 2c2132a6049c4ca1b1b667a07f80a007 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8fea8b6b7a1a4b01acbb4a2bb73d7541 (127.6.29.131:44627), 72a50bf0c64f4315be657e2a9031e6a0 (127.6.29.129:41513)
I20260925 04:56:47.875234 18118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:56:47.868742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:46503 (local address 127.6.29.131:44627)
0925 04:56:47.868974 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 04:56:47.868982 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:56:47.869786 (+ 804us) server_negotiation.cc:407] Connection header received
0925 04:56:47.869973 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:56:47.869980 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:56:47.870106 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:56:47.870212 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:56:47.870805 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:56:47.871514 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:56:47.873455 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:56:47.874236 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:56:47.874429 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:56:47.874509 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 04:56:47.874807 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 04:56:47.874945 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 04:56:47.875250 18117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:56:47.868832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:36989 (local address 127.6.29.129:41513)
0925 04:56:47.869078 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 04:56:47.869087 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:56:47.869914 (+ 827us) server_negotiation.cc:407] Connection header received
0925 04:56:47.870158 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:56:47.870165 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:56:47.870251 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:56:47.870373 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:56:47.871038 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:56:47.871740 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:56:47.873520 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:56:47.874303 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:56:47.874487 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:56:47.874562 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 04:56:47.874871 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 04:56:47.874997 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 04:56:47.875415 18131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:56:47.868770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:41513 (local address 127.6.29.130:36989)
0925 04:56:47.869780 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0925 04:56:47.869798 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:56:47.869958 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:56:47.870449 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:56:47.870457 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:56:47.870484 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:56:47.870892 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:56:47.870901 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:56:47.871946 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:56:47.871951 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:56:47.873372 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:56:47.873383 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:56:47.874526 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:56:47.874532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:56:47.874712 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:56:47.874733 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 04:56:47.874836 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 04:56:47.874971 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":889,"mutex_wait_us":34,"thread_start_us":565,"threads_started":1}
I20260925 04:56:47.875959 18130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:56:47.868626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.131:44627 (local address 127.6.29.130:46503)
0925 04:56:47.869670 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0925 04:56:47.869687 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:56:47.869815 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:56:47.870258 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:56:47.870265 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:56:47.870290 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:56:47.870642 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:56:47.870655 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:56:47.871645 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:56:47.871653 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:56:47.873298 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:56:47.873311 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:56:47.874473 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:56:47.874480 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:56:47.874648 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:56:47.874673 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 04:56:47.874782 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 04:56:47.874923 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":894,"mutex_wait_us":4,"thread_start_us":544,"threads_started":1}
I20260925 04:56:47.877113 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56df987f276749098b0e06659c362a03" candidate_uuid: "2c2132a6049c4ca1b1b667a07f80a007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72a50bf0c64f4315be657e2a9031e6a0" is_pre_election: true
I20260925 04:56:47.877400 18057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56df987f276749098b0e06659c362a03" candidate_uuid: "2c2132a6049c4ca1b1b667a07f80a007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fea8b6b7a1a4b01acbb4a2bb73d7541" is_pre_election: true
I20260925 04:56:47.877681 17780 raft_consensus.cc:2454] T 56df987f276749098b0e06659c362a03 P 72a50bf0c64f4315be657e2a9031e6a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c2132a6049c4ca1b1b667a07f80a007 in term 0.
I20260925 04:56:47.877796 18057 raft_consensus.cc:2454] T 56df987f276749098b0e06659c362a03 P 8fea8b6b7a1a4b01acbb4a2bb73d7541 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c2132a6049c4ca1b1b667a07f80a007 in term 0.
I20260925 04:56:47.878408 17857 leader_election.cc:302] T 56df987f276749098b0e06659c362a03 P 2c2132a6049c4ca1b1b667a07f80a007 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2c2132a6049c4ca1b1b667a07f80a007, 72a50bf0c64f4315be657e2a9031e6a0, 8fea8b6b7a1a4b01acbb4a2bb73d7541; no voters:
I20260925 04:56:47.878902 18127 raft_consensus.cc:2781] T 56df987f276749098b0e06659c362a03 P 2c2132a6049c4ca1b1b667a07f80a007 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:56:47.879084 18127 raft_consensus.cc:482] T 56df987f276749098b0e06659c362a03 P 2c2132a6049c4ca1b1b667a07f80a007 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:56:47.879197 18127 raft_consensus.cc:3037] T 56df987f276749098b0e06659c362a03 P 2c2132a6049c4ca1b1b667a07f80a007 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:56:47.881465 18127 raft_consensus.cc:504] T 56df987f276749098b0e06659c362a03 P 2c2132a6049c4ca1b1b667a07f80a007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fea8b6b7a1a4b01acbb4a2bb73d7541" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 44627 } } peers { permanent_uuid: "72a50bf0c64f4315be657e2a9031e6a0" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41513 } } peers { permanent_uuid: "2c2132a6049c4ca1b1b667a07f80a007" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44493 } }
I20260925 04:56:47.882076 18127 leader_election.cc:288] T 56df987f276749098b0e06659c362a03 P 2c2132a6049c4ca1b1b667a07f80a007 [CANDIDATE]: Term 1 election: Requested vote from peers 8fea8b6b7a1a4b01acbb4a2bb73d7541 (127.6.29.131:44627), 72a50bf0c64f4315be657e2a9031e6a0 (127.6.29.129:41513)
I20260925 04:56:47.882651 17780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56df987f276749098b0e06659c362a03" candidate_uuid: "2c2132a6049c4ca1b1b667a07f80a007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72a50bf0c64f4315be657e2a9031e6a0"
I20260925 04:56:47.882650 18057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56df987f276749098b0e06659c362a03" candidate_uuid: "2c2132a6049c4ca1b1b667a07f80a007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fea8b6b7a1a4b01acbb4a2bb73d7541"
I20260925 04:56:47.882874 17780 raft_consensus.cc:3037] T 56df987f276749098b0e06659c362a03 P 72a50bf0c64f4315be657e2a9031e6a0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:56:47.882874 18057 raft_consensus.cc:3037] T 56df987f276749098b0e06659c362a03 P 8fea8b6b7a1a4b01acbb4a2bb73d7541 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:56:47.886188 17780 raft_consensus.cc:2454] T 56df987f276749098b0e06659c362a03 P 72a50bf0c64f4315be657e2a9031e6a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c2132a6049c4ca1b1b667a07f80a007 in term 1.
I20260925 04:56:47.886188 18057 raft_consensus.cc:2454] T 56df987f276749098b0e06659c362a03 P 8fea8b6b7a1a4b01acbb4a2bb73d7541 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c2132a6049c4ca1b1b667a07f80a007 in term 1.
I20260925 04:56:47.886677 17857 leader_election.cc:302] T 56df987f276749098b0e06659c362a03 P 2c2132a6049c4ca1b1b667a07f80a007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c2132a6049c4ca1b1b667a07f80a007, 72a50bf0c64f4315be657e2a9031e6a0; no voters:
I20260925 04:56:47.887027 18127 raft_consensus.cc:2781] T 56df987f276749098b0e06659c362a03 P 2c2132a6049c4ca1b1b667a07f80a007 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:56:47.887923 18127 raft_consensus.cc:686] T 56df987f276749098b0e06659c362a03 P 2c2132a6049c4ca1b1b667a07f80a007 [term 1 LEADER]: Becoming Leader. State: Replica: 2c2132a6049c4ca1b1b667a07f80a007, State: Running, Role: LEADER
I20260925 04:56:47.888509 18127 consensus_queue.cc:237] T 56df987f276749098b0e06659c362a03 P 2c2132a6049c4ca1b1b667a07f80a007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fea8b6b7a1a4b01acbb4a2bb73d7541" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 44627 } } peers { permanent_uuid: "72a50bf0c64f4315be657e2a9031e6a0" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41513 } } peers { permanent_uuid: "2c2132a6049c4ca1b1b667a07f80a007" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44493 } }
I20260925 04:56:47.893460 17609 catalog_manager.cc:5495] T 56df987f276749098b0e06659c362a03 P 2c2132a6049c4ca1b1b667a07f80a007 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c2132a6049c4ca1b1b667a07f80a007 (127.6.29.130). New cstate: current_term: 1 leader_uuid: "2c2132a6049c4ca1b1b667a07f80a007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fea8b6b7a1a4b01acbb4a2bb73d7541" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 44627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72a50bf0c64f4315be657e2a9031e6a0" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c2132a6049c4ca1b1b667a07f80a007" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44493 } health_report { overall_health: HEALTHY } } }
I20260925 04:56:47.923753 18119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:56:47.918058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42444 (local address 127.6.29.130:44493)
0925 04:56:47.918260 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 04:56:47.918268 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:56:47.918370 (+ 102us) server_negotiation.cc:407] Connection header received
0925 04:56:47.918631 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:56:47.918636 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:56:47.918728 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:56:47.918893 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:56:47.919287 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:56:47.920004 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:56:47.921508 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:56:47.921916 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:56:47.922062 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:56:47.922600 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:56:47.923149 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:56:47.923323 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 04:56:47.923378 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 04:56:47.923464 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 04:56:47.932137 18057 raft_consensus.cc:1261] T 56df987f276749098b0e06659c362a03 P 8fea8b6b7a1a4b01acbb4a2bb73d7541 [term 1 FOLLOWER]: Refusing update from remote peer 2c2132a6049c4ca1b1b667a07f80a007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:56:47.932137 17780 raft_consensus.cc:1261] T 56df987f276749098b0e06659c362a03 P 72a50bf0c64f4315be657e2a9031e6a0 [term 1 FOLLOWER]: Refusing update from remote peer 2c2132a6049c4ca1b1b667a07f80a007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:56:47.933154 18132 consensus_queue.cc:1035] T 56df987f276749098b0e06659c362a03 P 2c2132a6049c4ca1b1b667a07f80a007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fea8b6b7a1a4b01acbb4a2bb73d7541" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 44627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:56:47.933525 18127 consensus_queue.cc:1035] T 56df987f276749098b0e06659c362a03 P 2c2132a6049c4ca1b1b667a07f80a007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72a50bf0c64f4315be657e2a9031e6a0" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:56:47.945349 18135 mvcc.cc:205] Tried to move back new op lower bound from 7333118803679444992 to 7333118803515527168. Current Snapshot: MvccSnapshot[applied={T|T < 7333118803679444992}]
I20260925 04:56:47.951283 18137 mvcc.cc:205] Tried to move back new op lower bound from 7333118803679444992 to 7333118803515527168. Current Snapshot: MvccSnapshot[applied={T|T < 7333118803679444992}]
I20260925 04:56:47.964623 17609 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:56:47.964924 17609 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55198:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:56:47.968933 17609 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9d770fbff972447faf59483b053a4466: Sending DeleteTablet for 3 replicas of tablet 56df987f276749098b0e06659c362a03
I20260925 04:56:47.970355 17760 tablet_service.cc:1514] Processing DeleteTablet for tablet 56df987f276749098b0e06659c362a03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:56:47 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:44680
I20260925 04:56:47.970531 17903 tablet_service.cc:1514] Processing DeleteTablet for tablet 56df987f276749098b0e06659c362a03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:56:47 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:42434
I20260925 04:56:47.971150 18037 tablet_service.cc:1514] Processing DeleteTablet for tablet 56df987f276749098b0e06659c362a03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:56:47 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:32854
I20260925 04:56:47.971853 18146 tablet_replica.cc:330] T 56df987f276749098b0e06659c362a03 P 72a50bf0c64f4315be657e2a9031e6a0: stopping tablet replica
I20260925 04:56:47.972406 18146 raft_consensus.cc:2229] T 56df987f276749098b0e06659c362a03 P 72a50bf0c64f4315be657e2a9031e6a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:56:47.972852 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 17692
I20260925 04:56:47.973047 18146 raft_consensus.cc:2258] T 56df987f276749098b0e06659c362a03 P 72a50bf0c64f4315be657e2a9031e6a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:56:47.973850 18148 tablet_replica.cc:330] T 56df987f276749098b0e06659c362a03 P 8fea8b6b7a1a4b01acbb4a2bb73d7541: stopping tablet replica
I20260925 04:56:47.974270 18148 raft_consensus.cc:2229] T 56df987f276749098b0e06659c362a03 P 8fea8b6b7a1a4b01acbb4a2bb73d7541 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:56:47.974766 18148 raft_consensus.cc:2258] T 56df987f276749098b0e06659c362a03 P 8fea8b6b7a1a4b01acbb4a2bb73d7541 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:56:47.975365 18146 ts_tablet_manager.cc:1905] T 56df987f276749098b0e06659c362a03 P 72a50bf0c64f4315be657e2a9031e6a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:56:47.977049 18148 ts_tablet_manager.cc:1905] T 56df987f276749098b0e06659c362a03 P 8fea8b6b7a1a4b01acbb4a2bb73d7541: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:56:47.982867 18146 ts_tablet_manager.cc:1918] T 56df987f276749098b0e06659c362a03 P 72a50bf0c64f4315be657e2a9031e6a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:56:47.983115 18146 log.cc:1196] T 56df987f276749098b0e06659c362a03 P 72a50bf0c64f4315be657e2a9031e6a0: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-0/wal/wals/56df987f276749098b0e06659c362a03
Sep 25 04:56:47 dist-test-slave-xzjr krb5kdc[17135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.129@KRBTEST.COM
I20260925 04:56:47.987108 18147 tablet_replica.cc:330] T 56df987f276749098b0e06659c362a03 P 2c2132a6049c4ca1b1b667a07f80a007: stopping tablet replica
I20260925 04:56:47.987162 18148 ts_tablet_manager.cc:1918] T 56df987f276749098b0e06659c362a03 P 8fea8b6b7a1a4b01acbb4a2bb73d7541: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:56:47.987326 18148 log.cc:1196] T 56df987f276749098b0e06659c362a03 P 8fea8b6b7a1a4b01acbb4a2bb73d7541: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-2/wal/wals/56df987f276749098b0e06659c362a03
I20260925 04:56:47.987694 18147 raft_consensus.cc:2229] T 56df987f276749098b0e06659c362a03 P 2c2132a6049c4ca1b1b667a07f80a007 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:56:47.987919 18148 ts_tablet_manager.cc:1939] T 56df987f276749098b0e06659c362a03 P 8fea8b6b7a1a4b01acbb4a2bb73d7541: Deleting consensus metadata
I20260925 04:56:47.988498 18147 raft_consensus.cc:2258] T 56df987f276749098b0e06659c362a03 P 2c2132a6049c4ca1b1b667a07f80a007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:56:47.990022 17597 catalog_manager.cc:4841] TS 8fea8b6b7a1a4b01acbb4a2bb73d7541 (127.6.29.131:44627): tablet 56df987f276749098b0e06659c362a03 (table test-table [id=6809f77e0ac24ddaacefbad92ee2caf0]) successfully deleted
I20260925 04:56:47.983726 18146 ts_tablet_manager.cc:1939] T 56df987f276749098b0e06659c362a03 P 72a50bf0c64f4315be657e2a9031e6a0: Deleting consensus metadata
I20260925 04:56:47.991132 18147 ts_tablet_manager.cc:1905] T 56df987f276749098b0e06659c362a03 P 2c2132a6049c4ca1b1b667a07f80a007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:56:47.992617 17596 catalog_manager.cc:4841] TS 72a50bf0c64f4315be657e2a9031e6a0 (127.6.29.129:41513): tablet 56df987f276749098b0e06659c362a03 (table test-table [id=6809f77e0ac24ddaacefbad92ee2caf0]) successfully deleted
I20260925 04:56:47.996153 18117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:56:47.975712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44690 (local address 127.6.29.129:41513)
0925 04:56:47.975922 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 04:56:47.975928 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:56:47.975951 (+ 23us) server_negotiation.cc:407] Connection header received
0925 04:56:47.976016 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:56:47.976021 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:56:47.976112 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:56:47.976220 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:56:47.978009 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:56:47.979131 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:56:47.981197 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:56:47.981901 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:56:47.982001 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:56:47.988081 (+ 6080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:56:47.988098 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:56:47.988119 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:56:47.988170 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:56:47.992332 (+ 4162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:56:47.993321 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:56:47.993333 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:56:47.993339 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:56:47.993519 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:56:47.994008 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:56:47.994014 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:56:47.994020 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:56:47.994495 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:56:47.994788 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0925 04:56:47.995403 (+ 615us) server_negotiation.cc:299] Negotiation successful
0925 04:56:47.995688 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 04:56:47.997437 17820 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:56:47.997605 18147 ts_tablet_manager.cc:1918] T 56df987f276749098b0e06659c362a03 P 2c2132a6049c4ca1b1b667a07f80a007: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:56:47.997776 18147 log.cc:1196] T 56df987f276749098b0e06659c362a03 P 2c2132a6049c4ca1b1b667a07f80a007: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790312053666503-6262-0/minicluster-data/ts-1/wal/wals/56df987f276749098b0e06659c362a03
I20260925 04:56:47.998246 18147 ts_tablet_manager.cc:1939] T 56df987f276749098b0e06659c362a03 P 2c2132a6049c4ca1b1b667a07f80a007: Deleting consensus metadata
I20260925 04:56:48.000077 17596 catalog_manager.cc:4841] TS 2c2132a6049c4ca1b1b667a07f80a007 (127.6.29.130:44493): tablet 56df987f276749098b0e06659c362a03 (table test-table [id=6809f77e0ac24ddaacefbad92ee2caf0]) successfully deleted
I20260925 04:56:48.090461 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 17692
I20260925 04:56:48.109762 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 17831
Sep 25 04:56:48 dist-test-slave-xzjr krb5kdc[17135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.130@KRBTEST.COM
I20260925 04:56:48.129215 18119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:56:48.110537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42458 (local address 127.6.29.130:44493)
0925 04:56:48.110721 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 04:56:48.110729 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:56:48.110800 (+ 71us) server_negotiation.cc:407] Connection header received
0925 04:56:48.110997 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:56:48.111001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:56:48.111063 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:56:48.111166 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:56:48.112964 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:56:48.113829 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:56:48.115995 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:56:48.116530 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:56:48.116632 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:56:48.122124 (+ 5492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:56:48.122144 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:56:48.122166 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:56:48.122219 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:56:48.126212 (+ 3993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:56:48.126967 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:56:48.126979 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:56:48.126983 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:56:48.127170 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:56:48.127557 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:56:48.127564 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:56:48.127567 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:56:48.128072 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:56:48.128404 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0925 04:56:48.128566 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 04:56:48.128786 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 04:56:48.129949 17963 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:56:48.245630 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 17831
I20260925 04:56:48.264087 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 17974
Sep 25 04:56:48 dist-test-slave-xzjr krb5kdc[17135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.131@KRBTEST.COM
I20260925 04:56:48.279609 18118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:56:48.264921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32864 (local address 127.6.29.131:44627)
0925 04:56:48.265109 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 04:56:48.265114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:56:48.265135 (+ 21us) server_negotiation.cc:407] Connection header received
0925 04:56:48.265348 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:56:48.265353 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:56:48.265418 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:56:48.265510 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:56:48.266878 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:56:48.267581 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:56:48.269146 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:56:48.269538 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:56:48.269648 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:56:48.274343 (+ 4695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:56:48.274359 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:56:48.274376 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:56:48.274419 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:56:48.277174 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:56:48.277859 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:56:48.277866 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:56:48.277868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:56:48.277996 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:56:48.278322 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:56:48.278325 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:56:48.278328 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:56:48.278700 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:56:48.278904 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 04:56:48.279182 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 04:56:48.279339 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 04:56:48.280342 18097 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:56:48.370996 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 17974
I20260925 04:56:48.387462 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 17579
I20260925 04:56:48.388722 17641 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:56:48.501922 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 17579
I20260925 04:56:48.524293 6262 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 04:56:48.859838 6262 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 04:56:48.860086 6262 mini_ranger.cc:66] Stopping Ranger...
I20260925 04:56:49.206014 6262 mini_ranger.cc:68] Stopped Ranger
2026-09-25 04:56:49.206 UTC [17165] LOG: received smart shutdown request
2026-09-25 04:56:49.209 UTC [17165] LOG: background worker "logical replication launcher" (PID 17174) exited with exit code 1
2026-09-25 04:56:49.210 UTC [17169] LOG: shutting down
2026-09-25 04:56:49.384 UTC [17165] LOG: database system is shut down
2026-09-25T04:56:49Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50976 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:56:49 dist-test-slave-xzjr krb5kdc[18162](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:56:49 dist-test-slave-xzjr krb5kdc[18162](info): set up 2 sockets
Sep 25 04:56:49 dist-test-slave-xzjr krb5kdc[18162](info): commencing operation
krb5kdc: starting...
W20260925 04:56:50.624557 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.005s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 04:56:50 dist-test-slave-xzjr krb5kdc[18162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:56:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:56:50Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.29.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.29.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.29.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/rangeradmin_127.6.29.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.29.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/rangeradmin_127.6.29.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.29.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.29.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.29.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/rangerlookup_127.6.29.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.29.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/rangerlookup_127.6.29.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.29.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.29.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.29.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/HTTP_127.6.29.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.29.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/HTTP_127.6.29.148@KRBTEST.COM.keytab.
I20260925 04:56:50.958618 6262 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskjC0Yza/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/postgres -l logfile start
2026-09-25 04:56:53.587 UTC [18192] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 04:56:53.588 UTC [18192] LOG: listening on IPv4 address "127.6.29.148", port 59353
2026-09-25 04:56:53.591 UTC [18192] LOG: listening on Unix socket "/tmp/.s.PGSQL.59353"
2026-09-25 04:56:53.616 UTC [18195] LOG: database system was shut down at 2026-09-25 04:56:52 UTC
2026-09-25 04:56:53.620 UTC [18192] LOG: database system is ready to accept connections
I20260925 04:56:54.596302 6262 mini_postgres.cc:96] Postgres bound to 59353
2026-09-25 04:56:54.613 UTC [18203] FATAL: database "slave" does not exist
127.6.29.148:59353 - accepting connections
I20260925 04:56:54.613850 6262 mini_ranger.cc:161] Starting Ranger...
I20260925 04:56:54.642305 6262 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 04:56:55.148090 6262 mini_ranger.cc:87] Created ranger Postgres database
I20260925 04:56:55.148208 6262 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ranger-admin
2026-09-25 04:56:55,435 [I] DB FLAVOR :POSTGRES
2026-09-25 04:56:55,436 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 04:56:55,436 [I] Checking connection..
2026-09-25 04:56:55,436 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:59353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 04:56:55,674 [I] Checking connection passed.
2026-09-25 04:56:55,674 [I] --------- Verifying version history table ---------
2026-09-25 04:56:55,675 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:59353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 04:56:55,916 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 04:56:55,917 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:59353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 04:56:56,167 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 04:56:56,168 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 04:56:56,168 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:59353/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 04:56:56.399 UTC [18267] WARNING: there is no transaction in progress
2026-09-25 04:56:56,405 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 04:56:56,405 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:59353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 04:56:56,643 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 04:56:56,643 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 04:56:56,643 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:59353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 04:56:56,901 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:59353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 04:56:57,162 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:59353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-xzjr.c.gcp-upstream.internal','N') ;"
2026-09-25 04:56:57,420 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:59353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 04:56:57,669 [I] Table x_portal_user does not exist in database ranger
2026-09-25 04:56:57,670 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:59353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 04:56:57,938 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 04:56:57,939 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:59353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 04:56:58,170 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:59353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 04:56:58,439 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 04:56:58,439 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:59353/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 04:56:59.440 UTC [18375] WARNING: there is no transaction in progress
2026-09-25 04:56:59.457 UTC [18375] WARNING: there is no transaction in progress
2026-09-25 04:56:59.476 UTC [18375] WARNING: there is no transaction in progress
2026-09-25 04:56:59.493 UTC [18375] WARNING: there is no transaction in progress
2026-09-25 04:56:59.513 UTC [18375] WARNING: there is no transaction in progress
2026-09-25 04:56:59.535 UTC [18375] WARNING: there is no transaction in progress
2026-09-25 04:56:59.601 UTC [18375] WARNING: there is no transaction in progress
2026-09-25 04:56:59.608 UTC [18375] WARNING: there is no transaction in progress
2026-09-25 04:56:59.624 UTC [18375] WARNING: there is no transaction in progress
2026-09-25 04:56:59.637 UTC [18375] WARNING: there is no transaction in progress
2026-09-25 04:56:59.648 UTC [18375] WARNING: there is no transaction in progress
2026-09-25 04:56:59.664 UTC [18375] WARNING: there is no transaction in progress
2026-09-25 04:56:59.675 UTC [18375] WARNING: there is no transaction in progress
2026-09-25 04:56:59.682 UTC [18375] WARNING: there is no transaction in progress
2026-09-25 04:56:59.690 UTC [18375] WARNING: there is no transaction in progress
2026-09-25 04:56:59.697 UTC [18375] WARNING: there is no transaction in progress
2026-09-25 04:57:00,368 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 04:57:00,368 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:59353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 04:57:00,618 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 04:57:00,618 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:59353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 04:57:00,883 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 04:57:00,884 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:59353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 04:57:01,143 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 04:57:01,144 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:59353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 04:57:01,382 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:59353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 04:57:01,646 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:59353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-xzjr.c.gcp-upstream.internal';"
2026-09-25 04:57:01,905 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 04:57:01,905 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:59353/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 04:57:02,138 [I] DB_PATCHES have already been applied
I20260925 04:57:02.144460 6262 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ranger-admin:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskjC0Yza/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 04:57:02.144543 6262 mini_ranger.cc:193] Using host: 127.6.29.148
I20260925 04:57:02.159278 6262 mini_ranger.cc:227] Ranger admin URL: http://127.6.29.148:38433
Sep 25, 2026 4:57:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 4:57:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 4:57:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 4:57:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 4:57:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 4:57:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 4:57:03 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 04:57:03.385815 6262 mini_ranger.cc:160] Time spent starting Ranger: real 8.772s user 0.000s sys 0.040s
Sep 25, 2026 4:57:22 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 4:57:22 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 4:57:22 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 4:57:22 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 4:57:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 4:57:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 4:57:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 4:57:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 4:57:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 4:57:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 4:57:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 4:57:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 4:57:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 4:57:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 4:57:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 4:57:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 4:57:22 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 4:57:23 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 04:57:24.373824 6262 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.29.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.29.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.29.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/rangerkms_127.6.29.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.29.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/rangerkms_127.6.29.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.29.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.29.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.29.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/HTTP_127.6.29.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.29.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/HTTP_127.6.29.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 04:57:24 dist-test-slave-xzjr krb5kdc[18162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312244, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 04:57:24.540580 6262 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 04:57:24.561965 6262 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 04:57:25.557319 6262 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 04:57:25.557427 6262 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ranger-kms
I20260925 04:57:25.557487 6262 mini_ranger_kms.cc:227] Using postgres at 127.6.29.148:59353
2026-09-25 04:57:25,918 [I] DB FLAVOR :POSTGRES
2026-09-25 04:57:25,918 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 04:57:25,919 [I] Checking connection
2026-09-25 04:57:25,919 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:59353/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 04:57:26,156 [I] connection success
2026-09-25 04:57:26,156 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 04:57:26,156 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 04:57:26,156 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:59353/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 04:57:26,403 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 04:57:26,403 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 04:57:26,403 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 04:57:26,403 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.29.148:59353/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 04:57:26,678 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 04:57:27.221789 6262 mini_ranger_kms.cc:310] Created kms service
I20260925 04:57:27.426314 6262 mini_ranger_kms.cc:326] Created kudu user
I20260925 04:57:27.471693 6262 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 04:57:27 dist-test-slave-xzjr krb5kdc[18162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312244, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.29.148@KRBTEST.COM
I20260925 04:57:27.865300 6262 mini_ranger_kms.cc:384] Added ranger policy
I20260925 04:57:27.865631 6262 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ranger-kms:/tmp/dist-test-taskjC0Yza/build/asan/bin/postgresql.jar:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskjC0Yza/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 04:57:27.865684 6262 mini_ranger_kms.cc:240] Using host: 127.6.29.148
I20260925 04:57:27.884119 6262 mini_ranger_kms.cc:276] Ranger KMS PID: 18555
I20260925 04:57:27.884292 6262 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.29.148:34353
Sep 25, 2026 4:57:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 4:57:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 4:57:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 4:57:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 4:57:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 4:57:29 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 04:57:30.220938 6262 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.680s user 0.039s sys 0.329s
I20260925 04:57:30.221200 6262 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 04:57:30.221361 6262 mini_ranger_kms.cc:401] 127.6.29.148:34353/kms/v1/keys
Sep 25 04:57:30 dist-test-slave-xzjr krb5kdc[18162](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790312250, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.29.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 04:57:30 dist-test-slave-xzjr krb5kdc[18162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312250, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.29.148@KRBTEST.COM for HTTP/127.6.29.148@KRBTEST.COM
Sep 25 04:57:31 dist-test-slave-xzjr krb5kdc[18162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312250, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.29.148@KRBTEST.COM for HTTP/127.6.29.148@KRBTEST.COM
Sep 25 04:57:33 dist-test-slave-xzjr krb5kdc[18162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.29.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
I20260925 04:57:33.332192 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:39733
--webserver_interface=127.6.29.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:45735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:39733
--ranger_config_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.29.148:34353/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:57:33.661327 18606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:57:33.661708 18606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:57:33.661864 18606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:57:33.669404 18606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:57:33.669559 18606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:57:33.669663 18606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:57:33.669765 18606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:57:33.669847 18606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:57:33.680291 18606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:45735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.29.148:34353/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:39733
--ranger_config_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:39733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:57:33.681600 18606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:57:33.682920 18606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:57:33.695000 18611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:33.694952 18614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:33.694907 18612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:57:33.696189 18606 server_base.cc:1034] running on GCE node
I20260925 04:57:33.696973 18606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:57:33.698446 18606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:57:33.699687 18606 hybrid_clock.cc:648] HybridClock initialized: now 1790312253699648 us; error 54 us; skew 500 ppm
I20260925 04:57:33.700078 18606 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:57:33 dist-test-slave-xzjr krb5kdc[18162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312253, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:57:33.703564 18606 init.cc:377] Logged in from keytab as kudu/127.6.29.190@KRBTEST.COM (short username kudu)
I20260925 04:57:33.705209 18606 webserver.cc:460] Webserver started at http://127.6.29.190:43107/ using document root <none> and password file <none>
I20260925 04:57:33.705866 18606 fs_manager.cc:359] Metadata directory not provided
I20260925 04:57:33.705965 18606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:57:33.706218 18606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 04:57:33 dist-test-slave-xzjr krb5kdc[18162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312253, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for HTTP/127.6.29.148@KRBTEST.COM
I20260925 04:57:33.758114 18606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "87f5df83da2248e49d76f921eb4e52ff"
format_stamp: "Formatted at 2026-09-25 04:57:33 on dist-test-slave-xzjr"
server_key: "865fc6a11444f7da2ede809511647d6f"
server_key_iv: "d5bf39cecce857c7e51156c32ab69277"
server_key_version: "kuduclusterkey@0"
I20260925 04:57:33.759063 18606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "87f5df83da2248e49d76f921eb4e52ff"
format_stamp: "Formatted at 2026-09-25 04:57:33 on dist-test-slave-xzjr"
server_key: "865fc6a11444f7da2ede809511647d6f"
server_key_iv: "d5bf39cecce857c7e51156c32ab69277"
server_key_version: "kuduclusterkey@0"
I20260925 04:57:33.764858 18606 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 04:57:33.781337 18621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:33.783144 18606 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:57:33.783360 18606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "87f5df83da2248e49d76f921eb4e52ff"
format_stamp: "Formatted at 2026-09-25 04:57:33 on dist-test-slave-xzjr"
server_key: "865fc6a11444f7da2ede809511647d6f"
server_key_iv: "d5bf39cecce857c7e51156c32ab69277"
server_key_version: "kuduclusterkey@0"
I20260925 04:57:33.783612 18606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:57:33.800601 18606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:57:33.804874 18606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:57:33.805218 18606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:57:33.825887 18606 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:39733
I20260925 04:57:33.825914 18672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:39733 every 8 connection(s)
I20260925 04:57:33.827476 18606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:57:33.828591 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 18606
I20260925 04:57:33.828806 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 04:57:33.829623 6262 external_mini_cluster.cc:1428] Setting key ac75ec8b3e6eddf004f4aabf3b4e5745
I20260925 04:57:33.836800 18673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:57:33.851545 18673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff: Bootstrap starting.
Sep 25 04:57:33 dist-test-slave-xzjr krb5kdc[18162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:57:33.855784 18673 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff: Neither blocks nor log segments found. Creating new log.
I20260925 04:57:33.857116 18673 log.cc:824] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff: Log is configured to *not* fsync() on all Append() calls
I20260925 04:57:33.860702 18673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff: No bootstrap required, opened a new log
I20260925 04:57:33.861300 18676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:33.832125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36474 (local address 127.6.29.190:39733)
0925 04:57:33.836299 (+ 4174us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:33.836310 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:33.836353 (+ 43us) server_negotiation.cc:407] Connection header received
0925 04:57:33.838021 (+ 1668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:33.838044 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:33.838447 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:33.838834 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:33.840298 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:33.841631 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:33.842920 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:33.843590 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:33.843833 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:33.853515 (+ 9682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:33.853524 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:33.853540 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:33.853587 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:33.856749 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:33.857563 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:33.857570 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:33.857577 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:33.857825 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:33.858345 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:33.858350 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:33.858353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:33.858916 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:33.859228 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:33.859715 (+ 487us) server_negotiation.cc:299] Negotiation successful
0925 04:57:33.860171 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3933,"thread_start_us":3680,"threads_started":1}
I20260925 04:57:33.871531 18673 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f5df83da2248e49d76f921eb4e52ff" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39733 } }
I20260925 04:57:33.872072 18673 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:57:33.872184 18673 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87f5df83da2248e49d76f921eb4e52ff, State: Initialized, Role: FOLLOWER
I20260925 04:57:33.872840 18673 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f5df83da2248e49d76f921eb4e52ff" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39733 } }
I20260925 04:57:33.873139 18673 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:57:33.873306 18673 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:57:33.873481 18673 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:57:33.876780 18673 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f5df83da2248e49d76f921eb4e52ff" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39733 } }
I20260925 04:57:33.877408 18673 leader_election.cc:302] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87f5df83da2248e49d76f921eb4e52ff; no voters:
I20260925 04:57:33.878105 18673 leader_election.cc:288] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:57:33.878172 18678 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:57:33.879612 18678 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff [term 1 LEADER]: Becoming Leader. State: Replica: 87f5df83da2248e49d76f921eb4e52ff, State: Running, Role: LEADER
I20260925 04:57:33.880224 18678 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f5df83da2248e49d76f921eb4e52ff" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39733 } }
I20260925 04:57:33.880445 18673 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:57:33.882864 18680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87f5df83da2248e49d76f921eb4e52ff. Latest consensus state: current_term: 1 leader_uuid: "87f5df83da2248e49d76f921eb4e52ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f5df83da2248e49d76f921eb4e52ff" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39733 } } }
I20260925 04:57:33.883270 18680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff [sys.catalog]: This master's current role is: LEADER
I20260925 04:57:33.884829 18679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87f5df83da2248e49d76f921eb4e52ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f5df83da2248e49d76f921eb4e52ff" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39733 } } }
I20260925 04:57:33.885187 18679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff [sys.catalog]: This master's current role is: LEADER
I20260925 04:57:33.895637 18685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:57:33.902626 18685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:57:33.914345 18685 catalog_manager.cc:1347] Generated new cluster ID: 7e533830ca43403db27398b8e8c2ff5b
I20260925 04:57:33.914551 18685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:57:33.931182 18673 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 04:57:33.933704 18685 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:57:33.935676 18685 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:57:33.954633 18685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff: Generated new TSK 0
I20260925 04:57:33.955991 18685 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.29.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab.
Entry for principal kudu/127.6.29.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab.
I20260925 04:57:34.816109 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.29.148:34353/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.129:0
--local_ip_for_outbound_sockets=127.6.29.129
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:39733
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:45735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 04:57:35 dist-test-slave-xzjr krb5kdc[18162](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790312255, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 04:57:35.280061 18718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:57:35.280589 18718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:57:35.280802 18718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:57:35.292819 18718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:57:35.292990 18718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:57:35.293231 18718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.129
I20260925 04:57:35.309509 18718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:45735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.29.148:34353/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:39733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.129
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:57:35.311213 18718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:57:35.312646 18718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:57:35.337276 18728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:35.344377 18729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:35.349357 18731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:57:35.350514 18718 server_base.cc:1034] running on GCE node
I20260925 04:57:35.351401 18718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:57:35.354342 18718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:57:35.355654 18718 hybrid_clock.cc:648] HybridClock initialized: now 1790312255355591 us; error 73 us; skew 500 ppm
I20260925 04:57:35.356050 18718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:57:35 dist-test-slave-xzjr krb5kdc[18162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312255, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:57:35.359385 18718 init.cc:377] Logged in from keytab as kudu/127.6.29.129@KRBTEST.COM (short username kudu)
I20260925 04:57:35.374624 18718 webserver.cc:460] Webserver started at http://127.6.29.129:34857/ using document root <none> and password file <none>
I20260925 04:57:35.375543 18718 fs_manager.cc:359] Metadata directory not provided
I20260925 04:57:35.375712 18718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:57:35.376107 18718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 04:57:35 dist-test-slave-xzjr krb5kdc[18162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312255, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for HTTP/127.6.29.148@KRBTEST.COM
I20260925 04:57:35.399245 18718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-0/data/instance:
uuid: "c3fb4ba0ec384f07b318e85bb1ddf6ec"
format_stamp: "Formatted at 2026-09-25 04:57:35 on dist-test-slave-xzjr"
server_key: "3cda2edf1cfeedd13bb9b87561819c89"
server_key_iv: "315e4a068c3a9b5017a2d98e1ae1ef80"
server_key_version: "kuduclusterkey@0"
I20260925 04:57:35.400558 18718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance:
uuid: "c3fb4ba0ec384f07b318e85bb1ddf6ec"
format_stamp: "Formatted at 2026-09-25 04:57:35 on dist-test-slave-xzjr"
server_key: "3cda2edf1cfeedd13bb9b87561819c89"
server_key_iv: "315e4a068c3a9b5017a2d98e1ae1ef80"
server_key_version: "kuduclusterkey@0"
I20260925 04:57:35.410001 18718 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.004s
I20260925 04:57:35.439755 18738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:35.441797 18718 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.005s sys 0.002s
I20260925 04:57:35.442034 18718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-0/wal
uuid: "c3fb4ba0ec384f07b318e85bb1ddf6ec"
format_stamp: "Formatted at 2026-09-25 04:57:35 on dist-test-slave-xzjr"
server_key: "3cda2edf1cfeedd13bb9b87561819c89"
server_key_iv: "315e4a068c3a9b5017a2d98e1ae1ef80"
server_key_version: "kuduclusterkey@0"
I20260925 04:57:35.442332 18718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:57:35.467790 18718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:57:35.473326 18718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:57:35.473680 18718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:57:35.474905 18718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:57:35.476766 18718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:57:35.476866 18718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:35.476987 18718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:57:35.477052 18718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 04:57:35 dist-test-slave-xzjr krb5kdc[18162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312255, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for HTTP/127.6.29.148@KRBTEST.COM
I20260925 04:57:35.636214 18718 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.129:36947
I20260925 04:57:35.636257 18850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.129:36947 every 8 connection(s)
I20260925 04:57:35.637933 18718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
I20260925 04:57:35.640290 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 18718
I20260925 04:57:35.640476 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance
I20260925 04:57:35.641218 6262 external_mini_cluster.cc:1428] Setting key 16f004f536d4c7fb1193925f4babb6a3
Sep 25 04:57:35 dist-test-slave-xzjr krb5kdc[18162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312255, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.29.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.130@KRBTEST.COM" created.
I20260925 04:57:35.686609 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:35.653057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:36601 (local address 127.6.29.190:39733)
0925 04:57:35.666452 (+ 13395us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:35.666458 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:35.666484 (+ 26us) server_negotiation.cc:407] Connection header received
0925 04:57:35.668140 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:35.668146 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:35.668251 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:35.668363 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:35.670668 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:35.671877 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:35.674311 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:35.674773 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:35.674926 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:35.679368 (+ 4442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:35.679377 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:35.679385 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:35.679448 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:35.682596 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:35.683575 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:35.683584 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:35.683590 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:35.683748 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:35.684334 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:35.684342 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:35.684348 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:35.684743 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:35.684894 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:35.685844 (+ 950us) server_negotiation.cc:299] Negotiation successful
0925 04:57:35.686105 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13237,"thread_start_us":13414,"threads_started":1}
I20260925 04:57:35.687721 18855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:35.653771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39733 (local address 127.6.29.129:36601)
0925 04:57:35.664659 (+ 10888us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:35.664748 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0925 04:57:35.667562 (+ 2814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:57:35.668841 (+ 1279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:57:35.668855 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:57:35.669514 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:57:35.670444 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:35.670474 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:35.672071 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:35.672077 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:35.674130 (+ 2053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:35.674144 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:35.675017 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:35.675025 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:35.675346 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:35.676261 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:57:35.676298 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:57:35.679113 (+ 2815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:57:35.682790 (+ 3677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:35.682800 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:35.682820 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:35.683433 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:35.683906 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:35.683911 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:35.683914 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:35.684197 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:35.684904 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:57:35.684912 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:57:35.685386 (+ 474us) client_negotiation.cc:769] Sending connection context
0925 04:57:35.685714 (+ 328us) client_negotiation.cc:240] Negotiation successful
0925 04:57:35.686264 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10377,"thread_start_us":19623,"threads_started":1}
I20260925 04:57:35.689638 18851 heartbeater.cc:347] Connected to a master server at 127.6.29.190:39733
I20260925 04:57:35.689980 18851 heartbeater.cc:464] Registering TS with master...
I20260925 04:57:35.690831 18851 heartbeater.cc:511] Master 127.6.29.190:39733 requested a full tablet report, sending...
I20260925 04:57:35.693243 18638 ts_manager.cc:194] Registered new tserver with Master: c3fb4ba0ec384f07b318e85bb1ddf6ec (127.6.29.129:36947)
I20260925 04:57:35.694751 18638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.129@KRBTEST.COM'} at 127.6.29.129:36601
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab.
Entry for principal kudu/127.6.29.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab.
I20260925 04:57:35.721848 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.29.148:34353/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.130:0
--local_ip_for_outbound_sockets=127.6.29.130
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:39733
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:45735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 04:57:35 dist-test-slave-xzjr krb5kdc[18162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312255, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for HTTP/127.6.29.148@KRBTEST.COM
W20260925 04:57:36.186748 18857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:57:36.187127 18857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:57:36.187307 18857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:57:36.195007 18857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:57:36.195128 18857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:57:36.195328 18857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.130
I20260925 04:57:36.205545 18857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:45735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.29.148:34353/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:39733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.130
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:57:36.207020 18857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:57:36.208429 18857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:57:36.224792 18871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:36.224727 18872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:36.224774 18874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:57:36.229446 18857 server_base.cc:1034] running on GCE node
I20260925 04:57:36.230125 18857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:57:36.231204 18857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:57:36.232455 18857 hybrid_clock.cc:648] HybridClock initialized: now 1790312256232407 us; error 42 us; skew 500 ppm
I20260925 04:57:36.232831 18857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:57:36 dist-test-slave-xzjr krb5kdc[18162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312256, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:57:36.236259 18857 init.cc:377] Logged in from keytab as kudu/127.6.29.130@KRBTEST.COM (short username kudu)
I20260925 04:57:36.237844 18857 webserver.cc:460] Webserver started at http://127.6.29.130:36863/ using document root <none> and password file <none>
I20260925 04:57:36.238533 18857 fs_manager.cc:359] Metadata directory not provided
I20260925 04:57:36.238633 18857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:57:36.238915 18857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 04:57:36 dist-test-slave-xzjr krb5kdc[18162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312256, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for HTTP/127.6.29.148@KRBTEST.COM
I20260925 04:57:36.252805 18857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-1/data/instance:
uuid: "d69cbdc154b54c568ef3ea768f71dd9f"
format_stamp: "Formatted at 2026-09-25 04:57:36 on dist-test-slave-xzjr"
server_key: "8f17d11098ed91ecc7ba6e9afdebb971"
server_key_iv: "a4bdc30b788c3d180d767684ea47760b"
server_key_version: "kuduclusterkey@0"
I20260925 04:57:36.253610 18857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance:
uuid: "d69cbdc154b54c568ef3ea768f71dd9f"
format_stamp: "Formatted at 2026-09-25 04:57:36 on dist-test-slave-xzjr"
server_key: "8f17d11098ed91ecc7ba6e9afdebb971"
server_key_iv: "a4bdc30b788c3d180d767684ea47760b"
server_key_version: "kuduclusterkey@0"
I20260925 04:57:36.258999 18857 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 04:57:36.272863 18881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:36.274430 18857 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 04:57:36.274614 18857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-1/wal
uuid: "d69cbdc154b54c568ef3ea768f71dd9f"
format_stamp: "Formatted at 2026-09-25 04:57:36 on dist-test-slave-xzjr"
server_key: "8f17d11098ed91ecc7ba6e9afdebb971"
server_key_iv: "a4bdc30b788c3d180d767684ea47760b"
server_key_version: "kuduclusterkey@0"
I20260925 04:57:36.274854 18857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:57:36.294857 18857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:57:36.299113 18857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:57:36.299415 18857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:57:36.300549 18857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:57:36.302060 18857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:57:36.302172 18857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:36.302323 18857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:57:36.302397 18857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:36.346840 18857 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.130:36597
I20260925 04:57:36.346851 18993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.130:36597 every 8 connection(s)
I20260925 04:57:36.348347 18857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
I20260925 04:57:36.354157 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 18857
I20260925 04:57:36.354406 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance
I20260925 04:57:36.354848 6262 external_mini_cluster.cc:1428] Setting key a53dfb3ab2c7bbc6ed9044b0d7c1935b
Sep 25 04:57:36 dist-test-slave-xzjr krb5kdc[18162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312256, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.29.131@KRBTEST.COM; defaulting to no policy
I20260925 04:57:36.380321 18997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:36.355424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:46905 (local address 127.6.29.190:39733)
0925 04:57:36.360207 (+ 4783us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:36.360216 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:36.360239 (+ 23us) server_negotiation.cc:407] Connection header received
0925 04:57:36.360302 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:36.360311 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:36.360407 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:36.360527 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:36.362552 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:36.363747 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:36.366255 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:36.367022 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:36.367161 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:36.372118 (+ 4957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:36.372126 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:36.372131 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:36.372182 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:36.375888 (+ 3706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:36.377055 (+ 1167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:36.377061 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:36.377065 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:36.377225 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:36.377805 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:36.377811 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:36.377814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:36.378258 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:36.378401 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:36.379518 (+ 1117us) server_negotiation.cc:299] Negotiation successful
0925 04:57:36.379872 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4631,"thread_start_us":4509,"threads_started":1}
I20260925 04:57:36.381484 18996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:36.354903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39733 (local address 127.6.29.130:46905)
0925 04:57:36.355864 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:36.355913 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 04:57:36.358290 (+ 2377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:57:36.360846 (+ 2556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:57:36.360860 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:57:36.361486 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:57:36.362330 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:36.362356 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:36.363983 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:36.363988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:36.366048 (+ 2060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:36.366064 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:36.367283 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:36.367291 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:36.367630 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:36.368631 (+ 1001us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:57:36.368662 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:57:36.371807 (+ 3145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:57:36.376101 (+ 4294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:36.376117 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:36.376140 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:36.376900 (+ 760us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:36.377372 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:36.377377 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:36.377382 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:36.377663 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:36.378464 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:57:36.378474 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:57:36.379037 (+ 563us) client_negotiation.cc:769] Sending connection context
0925 04:57:36.379386 (+ 349us) client_negotiation.cc:240] Negotiation successful
0925 04:57:36.380034 (+ 648us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":536,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.131@KRBTEST.COM" created.
I20260925 04:57:36.383360 18994 heartbeater.cc:347] Connected to a master server at 127.6.29.190:39733
I20260925 04:57:36.383826 18994 heartbeater.cc:464] Registering TS with master...
I20260925 04:57:36.384776 18994 heartbeater.cc:511] Master 127.6.29.190:39733 requested a full tablet report, sending...
I20260925 04:57:36.386587 18638 ts_manager.cc:194] Registered new tserver with Master: d69cbdc154b54c568ef3ea768f71dd9f (127.6.29.130:36597)
I20260925 04:57:36.387733 18638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.130@KRBTEST.COM'} at 127.6.29.130:46905
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab.
Entry for principal kudu/127.6.29.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab.
I20260925 04:57:36.412446 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.29.148:34353/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.131:0
--local_ip_for_outbound_sockets=127.6.29.131
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:39733
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:45735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:57:36.699182 18851 heartbeater.cc:503] Master 127.6.29.190:39733 was elected leader, sending a full tablet report...
W20260925 04:57:36.754704 19000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:57:36.755010 19000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:57:36.755194 19000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:57:36.762799 19000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:57:36.762912 19000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:57:36.763109 19000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.131
I20260925 04:57:36.773340 19000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:45735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.6.29.148:34353/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:39733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.131
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:57:36.774770 19000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:57:36.776185 19000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:57:36.790920 19005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:36.790854 19006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:36.790891 19008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:57:36.791752 19000 server_base.cc:1034] running on GCE node
I20260925 04:57:36.792503 19000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:57:36.793545 19000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:57:36.794778 19000 hybrid_clock.cc:648] HybridClock initialized: now 1790312256794728 us; error 68 us; skew 500 ppm
I20260925 04:57:36.795123 19000 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:57:36 dist-test-slave-xzjr krb5kdc[18162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312256, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:57:36.798496 19000 init.cc:377] Logged in from keytab as kudu/127.6.29.131@KRBTEST.COM (short username kudu)
I20260925 04:57:36.800092 19000 webserver.cc:460] Webserver started at http://127.6.29.131:42127/ using document root <none> and password file <none>
I20260925 04:57:36.800782 19000 fs_manager.cc:359] Metadata directory not provided
I20260925 04:57:36.800882 19000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:57:36.801136 19000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 04:57:36 dist-test-slave-xzjr krb5kdc[18162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312256, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for HTTP/127.6.29.148@KRBTEST.COM
I20260925 04:57:36.815886 19000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-2/data/instance:
uuid: "9c30a7d195dc40c394ba71c26efe310a"
format_stamp: "Formatted at 2026-09-25 04:57:36 on dist-test-slave-xzjr"
server_key: "ba9e9e7fdd8a853c34b8d452e9b9bff0"
server_key_iv: "87859fa52f92f5d23cc07fb61b7ebbab"
server_key_version: "kuduclusterkey@0"
I20260925 04:57:36.816768 19000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance:
uuid: "9c30a7d195dc40c394ba71c26efe310a"
format_stamp: "Formatted at 2026-09-25 04:57:36 on dist-test-slave-xzjr"
server_key: "ba9e9e7fdd8a853c34b8d452e9b9bff0"
server_key_iv: "87859fa52f92f5d23cc07fb61b7ebbab"
server_key_version: "kuduclusterkey@0"
I20260925 04:57:36.822353 19000 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 04:57:36.839728 19015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:36.841295 19000 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:57:36.841493 19000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-2/wal
uuid: "9c30a7d195dc40c394ba71c26efe310a"
format_stamp: "Formatted at 2026-09-25 04:57:36 on dist-test-slave-xzjr"
server_key: "ba9e9e7fdd8a853c34b8d452e9b9bff0"
server_key_iv: "87859fa52f92f5d23cc07fb61b7ebbab"
server_key_version: "kuduclusterkey@0"
I20260925 04:57:36.841725 19000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:57:36.856068 19000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:57:36.859745 19000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:57:36.860124 19000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:57:36.861073 19000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:57:36.862470 19000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:57:36.862576 19000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:36.862730 19000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:57:36.862814 19000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:36.896958 19000 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.131:38483
I20260925 04:57:36.897002 19127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.131:38483 every 8 connection(s)
I20260925 04:57:36.898439 19000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
I20260925 04:57:36.906113 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 19000
I20260925 04:57:36.906327 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance
I20260925 04:57:36.906802 6262 external_mini_cluster.cc:1428] Setting key 90b4b455f7a0af161e92fe78c39395da
Sep 25 04:57:36 dist-test-slave-xzjr krb5kdc[18162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312256, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:57:36.918779 19130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:36.901164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:55355 (local address 127.6.29.190:39733)
0925 04:57:36.901677 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:36.901682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:36.902573 (+ 891us) server_negotiation.cc:407] Connection header received
0925 04:57:36.904527 (+ 1954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:36.904532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:36.904643 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:36.904764 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:36.906272 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:36.907095 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:36.908987 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:36.909680 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:36.909838 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:36.913126 (+ 3288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:36.913132 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:36.913135 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:36.913171 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:36.915970 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:36.916764 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:36.916769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:36.916771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:36.916906 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:36.917302 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:36.917305 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:36.917307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:36.917572 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:36.917680 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:36.918346 (+ 666us) server_negotiation.cc:299] Negotiation successful
0925 04:57:36.918516 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":251,"threads_started":1}
I20260925 04:57:36.919734 19131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:36.901658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39733 (local address 127.6.29.131:55355)
0925 04:57:36.902389 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:36.902434 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 04:57:36.904229 (+ 1795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:57:36.905015 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:57:36.905024 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:57:36.905463 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:57:36.906101 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:36.906118 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:36.907254 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:36.907259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:36.908803 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:36.908817 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:36.909919 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:36.909925 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:36.910141 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:36.910787 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:57:36.910809 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:57:36.912902 (+ 2093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:57:36.916111 (+ 3209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:36.916117 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:36.916131 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:36.916588 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:36.917003 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:36.917006 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:36.917008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:36.917173 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:36.917678 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:57:36.917683 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:57:36.917989 (+ 306us) client_negotiation.cc:769] Sending connection context
0925 04:57:36.918204 (+ 215us) client_negotiation.cc:240] Negotiation successful
0925 04:57:36.918603 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":305,"threads_started":1}
I20260925 04:57:36.921221 19128 heartbeater.cc:347] Connected to a master server at 127.6.29.190:39733
I20260925 04:57:36.921566 19128 heartbeater.cc:464] Registering TS with master...
I20260925 04:57:36.922463 19128 heartbeater.cc:511] Master 127.6.29.190:39733 requested a full tablet report, sending...
I20260925 04:57:36.924017 18638 ts_manager.cc:194] Registered new tserver with Master: 9c30a7d195dc40c394ba71c26efe310a (127.6.29.131:38483)
I20260925 04:57:36.924875 18638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.131@KRBTEST.COM'} at 127.6.29.131:55355
I20260925 04:57:36.935076 6262 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:57:36.936802 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 18718
Sep 25 04:57:36 dist-test-slave-xzjr krb5kdc[18162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.129@KRBTEST.COM
I20260925 04:57:36.955583 19133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:36.937923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51994 (local address 127.6.29.129:36947)
0925 04:57:36.938617 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:36.938625 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:36.938651 (+ 26us) server_negotiation.cc:407] Connection header received
0925 04:57:36.938707 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:36.938713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:36.938943 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:36.939075 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:36.940626 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:36.941720 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:36.943367 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:36.944097 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:36.944211 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:36.949478 (+ 5267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:36.949491 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:36.949511 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:36.949552 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:36.952707 (+ 3155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:36.953491 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:36.953498 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:36.953501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:36.953650 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:36.954012 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:36.954018 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:36.954022 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:36.954375 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:36.954686 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:36.954960 (+ 274us) server_negotiation.cc:299] Negotiation successful
0925 04:57:36.955232 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":345,"threads_started":1}
I20260925 04:57:36.956899 18846 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:57:37.041545 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 18718
I20260925 04:57:37.057858 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 18857
Sep 25 04:57:37 dist-test-slave-xzjr krb5kdc[18162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.130@KRBTEST.COM
I20260925 04:57:37.079480 19135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:37.058812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43210 (local address 127.6.29.130:36597)
0925 04:57:37.059322 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:37.059329 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:37.059353 (+ 24us) server_negotiation.cc:407] Connection header received
0925 04:57:37.059412 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:37.059419 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:37.059646 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:37.059843 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:37.061161 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:37.062162 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:37.063571 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:37.064225 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:37.064375 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:37.073269 (+ 8894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:37.073285 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:37.073301 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:37.073345 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:37.076348 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:37.077180 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:37.077188 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:37.077190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:37.077330 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:37.077721 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:37.077727 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:37.077730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:37.078158 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:37.078512 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:37.078806 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 04:57:37.079083 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":260,"threads_started":1}
I20260925 04:57:37.080955 18989 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:57:37.164943 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 18857
I20260925 04:57:37.181010 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 19000
Sep 25 04:57:37 dist-test-slave-xzjr krb5kdc[18162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.131@KRBTEST.COM
I20260925 04:57:37.197728 19137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:37.182030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60728 (local address 127.6.29.131:38483)
0925 04:57:37.182646 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:37.182654 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:37.182683 (+ 29us) server_negotiation.cc:407] Connection header received
0925 04:57:37.182740 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:37.182746 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:37.182970 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:37.183146 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:37.184520 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:37.185657 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:37.186915 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:37.187564 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:37.187703 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:37.192209 (+ 4506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:37.192220 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:37.192235 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:37.192275 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:37.194915 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:37.195603 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:37.195611 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:37.195613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:37.195737 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:37.196132 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:37.196138 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:37.196141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:37.196481 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:37.196794 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:37.197079 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 04:57:37.197349 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":316,"threads_started":1}
I20260925 04:57:37.199013 19123 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:57:37.282635 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 19000
I20260925 04:57:37.298821 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 18606
I20260925 04:57:37.300113 18668 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:57:37.406745 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 18606
I20260925 04:57:37.431284 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:39733
--webserver_interface=127.6.29.190
--webserver_port=43107
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:45735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:39733
--ranger_config_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.29.148:34353/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.29.148:34353/kms with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:57:37.759663 19140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:57:37.760090 19140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:57:37.760265 19140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:57:37.767700 19140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:57:37.767884 19140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:57:37.767976 19140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:57:37.768059 19140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:57:37.768131 19140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:57:37.778602 19140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:45735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.29.148:34353/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:39733
--ranger_config_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:39733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=43107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:57:37.780045 19140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:57:37.781412 19140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:57:37.793819 19145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:37.793913 19148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:37.793852 19146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:57:37.794868 19140 server_base.cc:1034] running on GCE node
I20260925 04:57:37.795711 19140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:57:37.797310 19140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:57:37.798549 19140 hybrid_clock.cc:648] HybridClock initialized: now 1790312257798496 us; error 69 us; skew 500 ppm
I20260925 04:57:37.798907 19140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:57:37 dist-test-slave-xzjr krb5kdc[18162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312257, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:57:37.802153 19140 init.cc:377] Logged in from keytab as kudu/127.6.29.190@KRBTEST.COM (short username kudu)
I20260925 04:57:37.803629 19140 webserver.cc:460] Webserver started at http://127.6.29.190:43107/ using document root <none> and password file <none>
I20260925 04:57:37.804371 19140 fs_manager.cc:359] Metadata directory not provided
I20260925 04:57:37.804522 19140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 04:57:37 dist-test-slave-xzjr krb5kdc[18162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312257, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for HTTP/127.6.29.148@KRBTEST.COM
I20260925 04:57:37.856904 19140 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 04:57:37.860685 19156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:37.862571 19140 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:57:37.862805 19140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "87f5df83da2248e49d76f921eb4e52ff"
format_stamp: "Formatted at 2026-09-25 04:57:33 on dist-test-slave-xzjr"
server_key: "865fc6a11444f7da2ede809511647d6f"
server_key_iv: "d5bf39cecce857c7e51156c32ab69277"
server_key_version: "kuduclusterkey@0"
I20260925 04:57:37.864285 19140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:57:37.894248 19140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:57:37.898612 19140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:57:37.898862 19140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:57:37.917764 19140 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:39733
I20260925 04:57:37.917771 19207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:39733 every 8 connection(s)
I20260925 04:57:37.919315 19140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:57:37.924430 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 19140
I20260925 04:57:37.925257 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.29.148:34353/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.129:36947
--local_ip_for_outbound_sockets=127.6.29.129
--tserver_master_addrs=127.6.29.190:39733
--webserver_port=34857
--webserver_interface=127.6.29.129
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:45735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.29.148:34353/kms with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:57:37.926247 19208 sys_catalog.cc:263] Verifying existing consensus state
I20260925 04:57:37.929527 19208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff: Bootstrap starting.
I20260925 04:57:37.948192 19208 log.cc:824] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff: Log is configured to *not* fsync() on all Append() calls
I20260925 04:57:37.957453 19208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:57:37.957967 19208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff: Bootstrap complete.
I20260925 04:57:37.966733 19208 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f5df83da2248e49d76f921eb4e52ff" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39733 } }
I20260925 04:57:37.967600 19208 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87f5df83da2248e49d76f921eb4e52ff, State: Initialized, Role: FOLLOWER
I20260925 04:57:37.968312 19208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f5df83da2248e49d76f921eb4e52ff" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39733 } }
I20260925 04:57:37.968551 19208 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:57:37.968675 19208 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:57:37.968906 19208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff [term 1 FOLLOWER]: Advancing to term 2
I20260925 04:57:37.973053 19208 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f5df83da2248e49d76f921eb4e52ff" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39733 } }
I20260925 04:57:37.973542 19208 leader_election.cc:302] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87f5df83da2248e49d76f921eb4e52ff; no voters:
I20260925 04:57:37.974179 19208 leader_election.cc:288] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 04:57:37.974323 19212 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff [term 2 FOLLOWER]: Leader election won for term 2
I20260925 04:57:37.975697 19212 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff [term 2 LEADER]: Becoming Leader. State: Replica: 87f5df83da2248e49d76f921eb4e52ff, State: Running, Role: LEADER
I20260925 04:57:37.976346 19212 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f5df83da2248e49d76f921eb4e52ff" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39733 } }
I20260925 04:57:37.976557 19208 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:57:37.978848 19214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87f5df83da2248e49d76f921eb4e52ff. Latest consensus state: current_term: 2 leader_uuid: "87f5df83da2248e49d76f921eb4e52ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f5df83da2248e49d76f921eb4e52ff" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39733 } } }
I20260925 04:57:37.979228 19214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff [sys.catalog]: This master's current role is: LEADER
I20260925 04:57:37.979789 19213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "87f5df83da2248e49d76f921eb4e52ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87f5df83da2248e49d76f921eb4e52ff" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39733 } } }
I20260925 04:57:37.980082 19213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff [sys.catalog]: This master's current role is: LEADER
I20260925 04:57:37.989712 19219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:57:37.995865 19219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:57:37.996763 19208 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 04:57:37.997339 19219 catalog_manager.cc:1259] Loaded cluster ID: 7e533830ca43403db27398b8e8c2ff5b
I20260925 04:57:37.997428 19219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:57:38.001538 19219 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:57:38.003109 19219 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff: Loaded TSK: 0
I20260925 04:57:38.004714 19219 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 04:57:38.386782 19210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:57:38.387279 19210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:57:38.387483 19210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:57:38.399263 19210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:57:38.399468 19210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:57:38.399693 19210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.129
I20260925 04:57:38.415664 19210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:45735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.29.148:34353/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.129:36947
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.29.129
--webserver_port=34857
--tserver_master_addrs=127.6.29.190:39733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.129
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:57:38.417507 19210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:57:38.419351 19210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:57:38.446708 19245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:38.451254 19246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:57:38.453804 19210 server_base.cc:1034] running on GCE node
W20260925 04:57:38.453994 19249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:57:38.455006 19210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:57:38.458057 19210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:57:38.459395 19210 hybrid_clock.cc:648] HybridClock initialized: now 1790312258459319 us; error 75 us; skew 500 ppm
I20260925 04:57:38.459849 19210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:57:38 dist-test-slave-xzjr krb5kdc[18162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312258, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:57:38.463891 19210 init.cc:377] Logged in from keytab as kudu/127.6.29.129@KRBTEST.COM (short username kudu)
I20260925 04:57:38.466938 19210 webserver.cc:460] Webserver started at http://127.6.29.129:34857/ using document root <none> and password file <none>
I20260925 04:57:38.467752 19210 fs_manager.cc:359] Metadata directory not provided
I20260925 04:57:38.467988 19210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 04:57:38 dist-test-slave-xzjr krb5kdc[18162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312258, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for HTTP/127.6.29.148@KRBTEST.COM
I20260925 04:57:38.491999 19210 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 04:57:38.495934 19256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:38.497999 19210 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:57:38.498289 19210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-0/wal
uuid: "c3fb4ba0ec384f07b318e85bb1ddf6ec"
format_stamp: "Formatted at 2026-09-25 04:57:35 on dist-test-slave-xzjr"
server_key: "3cda2edf1cfeedd13bb9b87561819c89"
server_key_iv: "315e4a068c3a9b5017a2d98e1ae1ef80"
server_key_version: "kuduclusterkey@0"
I20260925 04:57:38.499898 19210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:57:38.575063 19210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:57:38.579746 19210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:57:38.580102 19210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:57:38.581234 19210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:57:38.584640 19210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:57:38.584779 19210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:38.584923 19210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:57:38.585031 19210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:38.680184 19210 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.129:36947
I20260925 04:57:38.682453 19210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
I20260925 04:57:38.680220 19369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.129:36947 every 8 connection(s)
I20260925 04:57:38.685076 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 19210
I20260925 04:57:38.685906 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.29.148:34353/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.130:36597
--local_ip_for_outbound_sockets=127.6.29.130
--tserver_master_addrs=127.6.29.190:39733
--webserver_port=36863
--webserver_interface=127.6.29.130
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:45735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.29.148:34353/kms with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 04:57:38 dist-test-slave-xzjr krb5kdc[18162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312258, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:57:38.732668 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:38.686448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39733 (local address 127.6.29.129:57859)
0925 04:57:38.697965 (+ 11517us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:38.698019 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 04:57:38.700937 (+ 2918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:57:38.703697 (+ 2760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:57:38.703712 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:57:38.704514 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:57:38.705420 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:38.705451 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:38.708115 (+ 2664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:38.708124 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:38.710524 (+ 2400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:38.710546 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:38.712868 (+ 2322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:38.712880 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:38.713229 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:38.714123 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:57:38.714164 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:57:38.717375 (+ 3211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:57:38.726169 (+ 8794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:38.726181 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:38.726199 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:38.726889 (+ 690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:38.727496 (+ 607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:38.727501 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:38.727504 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:38.727753 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:38.729712 (+ 1959us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:57:38.729721 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:57:38.730196 (+ 475us) client_negotiation.cc:769] Sending connection context
0925 04:57:38.730538 (+ 342us) client_negotiation.cc:240] Negotiation successful
0925 04:57:38.731166 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11244,"thread_start_us":11037,"threads_started":1}
I20260925 04:57:38.735359 19372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:38.686268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:57859 (local address 127.6.29.190:39733)
0925 04:57:38.697409 (+ 11141us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:38.697427 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:38.698192 (+ 765us) server_negotiation.cc:407] Connection header received
0925 04:57:38.701866 (+ 3674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:38.701889 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:38.702471 (+ 582us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:38.702923 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:38.705710 (+ 2787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:38.707885 (+ 2175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:38.710772 (+ 2887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:38.712147 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:38.712386 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:38.718920 (+ 6534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:38.718932 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:38.718957 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:38.719026 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:38.725938 (+ 6912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:38.727096 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:38.727107 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:38.727115 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:38.727346 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:38.727920 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:38.727925 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:38.727928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:38.728650 (+ 722us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:38.728837 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:38.733604 (+ 4767us) server_negotiation.cc:299] Negotiation successful
0925 04:57:38.734078 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10835,"thread_start_us":10601,"threads_started":1}
I20260925 04:57:38.738960 19370 heartbeater.cc:347] Connected to a master server at 127.6.29.190:39733
I20260925 04:57:38.739284 19370 heartbeater.cc:464] Registering TS with master...
I20260925 04:57:38.740358 19370 heartbeater.cc:511] Master 127.6.29.190:39733 requested a full tablet report, sending...
W20260925 04:57:38.744120 19370 heartbeater.cc:644] Failed to heartbeat to 127.6.29.190:39733 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 04:57:38.744403 19370 heartbeater.cc:464] Registering TS with master...
I20260925 04:57:38.744748 19370 heartbeater.cc:511] Master 127.6.29.190:39733 requested a full tablet report, sending...
I20260925 04:57:38.746148 19370 heartbeater.cc:464] Registering TS with master...
I20260925 04:57:38.746433 19370 heartbeater.cc:511] Master 127.6.29.190:39733 requested a full tablet report, sending...
W20260925 04:57:38.747555 19370 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 04:57:39.118432 19374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:57:39.118774 19374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:57:39.118945 19374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:57:39.127985 19374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:57:39.128136 19374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:57:39.128387 19374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.130
I20260925 04:57:39.144343 19374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:45735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.29.148:34353/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.130:36597
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.29.130
--webserver_port=36863
--tserver_master_addrs=127.6.29.190:39733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.130
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:57:39.146353 19374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:57:39.148272 19374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:57:39.177263 19391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:39.177420 19393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:39.184201 19389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:57:39.186337 19374 server_base.cc:1034] running on GCE node
I20260925 04:57:39.187419 19374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:57:39.195647 19374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:57:39.197077 19374 hybrid_clock.cc:648] HybridClock initialized: now 1790312259196955 us; error 118 us; skew 500 ppm
I20260925 04:57:39.197508 19374 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:57:39 dist-test-slave-xzjr krb5kdc[18162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312259, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:57:39.202071 19374 init.cc:377] Logged in from keytab as kudu/127.6.29.130@KRBTEST.COM (short username kudu)
I20260925 04:57:39.204178 19374 webserver.cc:460] Webserver started at http://127.6.29.130:36863/ using document root <none> and password file <none>
I20260925 04:57:39.205163 19374 fs_manager.cc:359] Metadata directory not provided
I20260925 04:57:39.205315 19374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 04:57:39 dist-test-slave-xzjr krb5kdc[18162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312259, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for HTTP/127.6.29.148@KRBTEST.COM
I20260925 04:57:39.250126 19374 fs_manager.cc:709] Time spent opening directory manager: real 0.021s user 0.011s sys 0.008s
I20260925 04:57:39.258725 19401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:39.261027 19374 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.003s sys 0.000s
Sep 25 04:57:39 dist-test-slave-xzjr krb5kdc[18162](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790312259, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:57:39.261277 19374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-1/wal
uuid: "d69cbdc154b54c568ef3ea768f71dd9f"
format_stamp: "Formatted at 2026-09-25 04:57:36 on dist-test-slave-xzjr"
server_key: "8f17d11098ed91ecc7ba6e9afdebb971"
server_key_iv: "a4bdc30b788c3d180d767684ea47760b"
server_key_version: "kuduclusterkey@0"
I20260925 04:57:39.262794 19374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:57:39.291286 19374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:57:39.296440 19374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:57:39.296860 19374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:57:39.299075 19374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:57:39.302388 19374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:57:39.302583 19374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260925 04:57:39.302750 19374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:57:39.302909 19374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:39.350417 19374 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.130:36597
I20260925 04:57:39.350461 19514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.130:36597 every 8 connection(s)
I20260925 04:57:39.352181 19374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
I20260925 04:57:39.354221 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 19374
I20260925 04:57:39.355041 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.29.148:34353/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.131:38483
--local_ip_for_outbound_sockets=127.6.29.131
--tserver_master_addrs=127.6.29.190:39733
--webserver_port=42127
--webserver_interface=127.6.29.131
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:45735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.29.148:34353/kms with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 04:57:39 dist-test-slave-xzjr krb5kdc[18162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312259, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:57:39.389060 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:39.356301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39733 (local address 127.6.29.130:33669)
0925 04:57:39.365945 (+ 9644us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:39.366018 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0925 04:57:39.368818 (+ 2800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:57:39.370236 (+ 1418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:57:39.370249 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:57:39.370758 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:57:39.371573 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:39.371593 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:39.372739 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:39.372751 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:39.374104 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:39.374126 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:39.375008 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:39.375018 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:39.375297 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:39.376177 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:57:39.376210 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:57:39.379239 (+ 3029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:57:39.383658 (+ 4419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:39.383671 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:39.383695 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:39.384459 (+ 764us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:39.385000 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:39.385005 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:39.385009 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:39.385283 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:39.386055 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:57:39.386065 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:57:39.386549 (+ 484us) client_negotiation.cc:769] Sending connection context
0925 04:57:39.386891 (+ 342us) client_negotiation.cc:240] Negotiation successful
0925 04:57:39.387524 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9319,"thread_start_us":8053,"threads_started":1}
I20260925 04:57:39.390260 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:39.356219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:33669 (local address 127.6.29.190:39733)
0925 04:57:39.363662 (+ 7443us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:39.363670 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:39.368624 (+ 4954us) server_negotiation.cc:407] Connection header received
0925 04:57:39.369243 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:39.369251 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:39.369352 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:39.369438 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:39.371804 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:39.372546 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:39.374284 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:39.374732 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:39.374887 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:39.379538 (+ 4651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:39.379544 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:39.379547 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:39.379584 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:39.383400 (+ 3816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:39.384667 (+ 1267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:39.384675 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:39.384680 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:39.384857 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:39.385426 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:39.385432 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:39.385435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:39.385811 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:39.385974 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:39.389593 (+ 3619us) server_negotiation.cc:299] Negotiation successful
0925 04:57:39.389873 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7282,"thread_start_us":7551,"threads_started":1}
I20260925 04:57:39.393035 19515 heartbeater.cc:347] Connected to a master server at 127.6.29.190:39733
I20260925 04:57:39.393411 19515 heartbeater.cc:464] Registering TS with master...
I20260925 04:57:39.394356 19515 heartbeater.cc:511] Master 127.6.29.190:39733 requested a full tablet report, sending...
I20260925 04:57:39.396734 19173 ts_manager.cc:194] Registered new tserver with Master: d69cbdc154b54c568ef3ea768f71dd9f (127.6.29.130:36597)
I20260925 04:57:39.398989 19173 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.130@KRBTEST.COM'} at 127.6.29.130:33669
Sep 25 04:57:39 dist-test-slave-xzjr krb5kdc[18162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312259, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for HTTP/127.6.29.148@KRBTEST.COM
I20260925 04:57:39.747741 19370 heartbeater.cc:464] Registering TS with master...
I20260925 04:57:39.748277 19370 heartbeater.cc:511] Master 127.6.29.190:39733 requested a full tablet report, sending...
I20260925 04:57:39.749506 19173 ts_manager.cc:194] Registered new tserver with Master: c3fb4ba0ec384f07b318e85bb1ddf6ec (127.6.29.129:36947)
I20260925 04:57:39.750422 19173 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.129@KRBTEST.COM'} at 127.6.29.129:57859
W20260925 04:57:39.814599 19519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:57:39.815124 19519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:57:39.815322 19519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:57:39.826984 19519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:57:39.827230 19519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:57:39.827453 19519 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.131
I20260925 04:57:39.843206 19519 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:45735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.29.148:34353/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.131:38483
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.29.131
--webserver_port=42127
--tserver_master_addrs=127.6.29.190:39733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.131
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:57:39.844973 19519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:57:39.846678 19519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:57:39.865100 19524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:39.870543 19525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:39.878772 19527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:57:39.879144 19519 server_base.cc:1034] running on GCE node
I20260925 04:57:39.879990 19519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:57:39.884079 19519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:57:39.885408 19519 hybrid_clock.cc:648] HybridClock initialized: now 1790312259885331 us; error 64 us; skew 500 ppm
I20260925 04:57:39.885812 19519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:57:39 dist-test-slave-xzjr krb5kdc[18162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312259, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:57:39.890044 19519 init.cc:377] Logged in from keytab as kudu/127.6.29.131@KRBTEST.COM (short username kudu)
I20260925 04:57:39.892179 19519 webserver.cc:460] Webserver started at http://127.6.29.131:42127/ using document root <none> and password file <none>
I20260925 04:57:39.893111 19519 fs_manager.cc:359] Metadata directory not provided
I20260925 04:57:39.893253 19519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 04:57:39 dist-test-slave-xzjr krb5kdc[18162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312259, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for HTTP/127.6.29.148@KRBTEST.COM
I20260925 04:57:39.929188 19519 fs_manager.cc:709] Time spent opening directory manager: real 0.018s user 0.006s sys 0.005s
I20260925 04:57:39.932976 19535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:39.934845 19519 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260925 04:57:39.935086 19519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-2/wal
uuid: "9c30a7d195dc40c394ba71c26efe310a"
format_stamp: "Formatted at 2026-09-25 04:57:36 on dist-test-slave-xzjr"
server_key: "ba9e9e7fdd8a853c34b8d452e9b9bff0"
server_key_iv: "87859fa52f92f5d23cc07fb61b7ebbab"
server_key_version: "kuduclusterkey@0"
I20260925 04:57:39.936609 19519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:57:39.981170 19519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:57:39.987128 19519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:57:39.987516 19519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:57:39.989068 19519 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:57:39.994001 19519 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:57:39.994127 19519 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:39.994232 19519 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:57:39.994306 19519 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 04:57:40 dist-test-slave-xzjr krb5kdc[18162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312259, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for HTTP/127.6.29.148@KRBTEST.COM
I20260925 04:57:40.246647 19519 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.131:38483
I20260925 04:57:40.246690 19648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.131:38483 every 8 connection(s)
I20260925 04:57:40.248533 19519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
I20260925 04:57:40.253610 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 19519
Sep 25 04:57:40 dist-test-slave-xzjr krb5kdc[18162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312259, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:57:40.332156 19655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:40.254636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:53177 (local address 127.6.29.190:39733)
0925 04:57:40.300475 (+ 45839us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:40.300486 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:40.303943 (+ 3457us) server_negotiation.cc:407] Connection header received
0925 04:57:40.312200 (+ 8257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:40.312212 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:40.312371 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:40.312480 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:40.316177 (+ 3697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:40.317357 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:40.320320 (+ 2963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:40.320791 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:40.320940 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:40.325070 (+ 4130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:40.325077 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:40.325081 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:40.325128 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:40.328597 (+ 3469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:40.329567 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:40.329572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:40.329574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:40.329692 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:40.330119 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:40.330123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:40.330131 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:40.330465 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:40.330615 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:40.331503 (+ 888us) server_negotiation.cc:299] Negotiation successful
0925 04:57:40.331722 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45630,"thread_start_us":46284,"threads_started":1}
I20260925 04:57:40.333443 19653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:40.253953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39733 (local address 127.6.29.131:53177)
0925 04:57:40.301882 (+ 47929us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:40.301972 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0925 04:57:40.310761 (+ 8789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:57:40.313279 (+ 2518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:57:40.313296 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:57:40.314196 (+ 900us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:57:40.315271 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:40.315320 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:40.318010 (+ 2690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:40.318017 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:40.320161 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:40.320181 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:40.321040 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:40.321049 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:40.321353 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:40.322202 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:57:40.322232 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:57:40.324782 (+ 2550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:57:40.328746 (+ 3964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:40.328763 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:40.328780 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:40.329196 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:40.329805 (+ 609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:40.329811 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:40.329815 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:40.329996 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:40.330589 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:57:40.330597 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:57:40.331021 (+ 424us) client_negotiation.cc:769] Sending connection context
0925 04:57:40.331375 (+ 354us) client_negotiation.cc:240] Negotiation successful
0925 04:57:40.332018 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47458,"thread_start_us":53614,"threads_started":1}
I20260925 04:57:40.334285 19656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:40.254906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36526 (local address 127.6.29.190:39733)
0925 04:57:40.300892 (+ 45986us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:40.300900 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:40.317493 (+ 16593us) server_negotiation.cc:407] Connection header received
0925 04:57:40.317544 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:40.317553 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:40.317633 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:40.317751 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:40.319931 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:40.321184 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:40.323083 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:40.323718 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:40.323878 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:40.327395 (+ 3517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:40.327402 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:40.327406 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:40.327465 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:40.330964 (+ 3499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:40.331837 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:40.331842 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:40.331846 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:40.332001 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:40.332723 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:40.332727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:40.332730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:40.333049 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:40.333302 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:40.333676 (+ 374us) server_negotiation.cc:299] Negotiation successful
0925 04:57:40.333903 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45860,"thread_start_us":46106,"threads_started":1}
I20260925 04:57:40.335286 19649 heartbeater.cc:347] Connected to a master server at 127.6.29.190:39733
I20260925 04:57:40.335611 19649 heartbeater.cc:464] Registering TS with master...
I20260925 04:57:40.336477 19649 heartbeater.cc:511] Master 127.6.29.190:39733 requested a full tablet report, sending...
I20260925 04:57:40.338069 19173 ts_manager.cc:194] Registered new tserver with Master: 9c30a7d195dc40c394ba71c26efe310a (127.6.29.131:38483)
I20260925 04:57:40.338919 19173 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.131@KRBTEST.COM'} at 127.6.29.131:53177
I20260925 04:57:40.339675 6262 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:57:40.365293 19656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:40.350095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36540 (local address 127.6.29.190:39733)
0925 04:57:40.350364 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:40.350374 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:40.352366 (+ 1992us) server_negotiation.cc:407] Connection header received
0925 04:57:40.352593 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:40.352598 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:40.352676 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:40.352786 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:40.354372 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:40.355143 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:40.356725 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:40.357148 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:40.357295 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:40.359871 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:40.359876 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:40.359879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:40.359913 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:40.362572 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:40.363305 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:40.363311 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:40.363315 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:40.363466 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:40.363871 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:40.363878 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:40.363881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:40.364157 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:40.364425 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:40.364746 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 04:57:40.364975 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 04:57:40.370496 19173 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36540:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:57:40.378098 19173 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:57:40.403664 19515 heartbeater.cc:503] Master 127.6.29.190:39733 was elected leader, sending a full tablet report...
I20260925 04:57:40.404851 19671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:40.392864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:36947 (local address 127.0.0.1:53818)
0925 04:57:40.397135 (+ 4271us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:40.397159 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 04:57:40.397278 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:57:40.398506 (+ 1228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:57:40.398510 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:57:40.398534 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:57:40.398937 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:40.398950 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:40.400921 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:40.400927 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:40.402356 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:40.402376 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:40.403812 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:40.403820 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:40.404011 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:40.404072 (+ 61us) client_negotiation.cc:769] Sending connection context
0925 04:57:40.404190 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 04:57:40.404386 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4115,"mutex_wait_us":26,"thread_start_us":1198,"threads_started":1}
I20260925 04:57:40.405673 19673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:40.393285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42466 (local address 127.6.29.131:38483)
0925 04:57:40.395552 (+ 2267us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:40.395564 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:40.395605 (+ 41us) server_negotiation.cc:407] Connection header received
0925 04:57:40.395725 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:40.395735 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:40.396117 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:40.396367 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:57:40.397787 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:40.399297 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:40.402337 (+ 3040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:40.403845 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:40.403966 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:40.404190 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:40.404956 (+ 766us) server_negotiation.cc:299] Negotiation successful
0925 04:57:40.405151 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2063,"thread_start_us":1921,"threads_started":1}
I20260925 04:57:40.406109 19674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:40.393527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53818 (local address 127.6.29.129:36947)
0925 04:57:40.397826 (+ 4299us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:40.397835 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:40.397864 (+ 29us) server_negotiation.cc:407] Connection header received
0925 04:57:40.397979 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:40.397986 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:40.398279 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:40.399126 (+ 847us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:57:40.399360 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:40.400779 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:40.402525 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:40.403539 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:40.405333 (+ 1794us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:40.405564 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:40.405626 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 04:57:40.405776 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4131,"thread_start_us":3967,"threads_started":1}
I20260925 04:57:40.406563 19672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:40.392690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.131:38483 (local address 127.0.0.1:42466)
0925 04:57:40.394589 (+ 1899us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:40.394638 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 04:57:40.394861 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:57:40.396965 (+ 2104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:57:40.396975 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:57:40.397007 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:57:40.397553 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:40.397564 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:40.399472 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:40.399476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:40.402142 (+ 2666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:40.402161 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:40.404348 (+ 2187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:40.404360 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:40.404554 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:40.404594 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 04:57:40.404682 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 04:57:40.404834 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1552,"mutex_wait_us":36,"thread_start_us":4153,"threads_started":1}
I20260925 04:57:40.408780 19676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:40.397798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:36597 (local address 127.0.0.1:42030)
0925 04:57:40.398559 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:40.398576 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:57:40.398700 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:57:40.399259 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:57:40.399262 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:57:40.399286 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:57:40.400393 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:40.400405 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:40.402334 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:40.402339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:40.404432 (+ 2093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:40.404444 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:40.407965 (+ 3521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:40.407971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:40.408148 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:40.408185 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 04:57:40.408290 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 04:57:40.408424 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":7,"thread_start_us":515,"threads_started":1}
I20260925 04:57:40.408787 19675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:40.397527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42030 (local address 127.6.29.130:36597)
0925 04:57:40.398142 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:40.398150 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:40.398685 (+ 535us) server_negotiation.cc:407] Connection header received
0925 04:57:40.398836 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:40.398843 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:40.399073 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:40.399249 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:57:40.400617 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:40.402162 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:40.405713 (+ 3551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:40.407722 (+ 2009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:40.407911 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:40.408122 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:40.408332 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 04:57:40.408480 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":383,"threads_started":1}
I20260925 04:57:40.410320 19305 tablet_service.cc:1467] Processing CreateTablet for tablet 3fbb415fd2514162b4af574453d8caf7 (DEFAULT_TABLE table=test-table [id=c99652aef6114b6b9a14d051f0101dae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:57:40.410320 19584 tablet_service.cc:1467] Processing CreateTablet for tablet 3fbb415fd2514162b4af574453d8caf7 (DEFAULT_TABLE table=test-table [id=c99652aef6114b6b9a14d051f0101dae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:57:40.411944 19305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fbb415fd2514162b4af574453d8caf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:57:40.411926 19584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fbb415fd2514162b4af574453d8caf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:57:40.412367 19449 tablet_service.cc:1467] Processing CreateTablet for tablet 3fbb415fd2514162b4af574453d8caf7 (DEFAULT_TABLE table=test-table [id=c99652aef6114b6b9a14d051f0101dae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:57:40.413587 19449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fbb415fd2514162b4af574453d8caf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:57:40.425897 19677 tablet_bootstrap.cc:492] T 3fbb415fd2514162b4af574453d8caf7 P d69cbdc154b54c568ef3ea768f71dd9f: Bootstrap starting.
I20260925 04:57:40.429104 19679 tablet_bootstrap.cc:492] T 3fbb415fd2514162b4af574453d8caf7 P 9c30a7d195dc40c394ba71c26efe310a: Bootstrap starting.
I20260925 04:57:40.429430 19678 tablet_bootstrap.cc:492] T 3fbb415fd2514162b4af574453d8caf7 P c3fb4ba0ec384f07b318e85bb1ddf6ec: Bootstrap starting.
I20260925 04:57:40.429987 19677 tablet_bootstrap.cc:654] T 3fbb415fd2514162b4af574453d8caf7 P d69cbdc154b54c568ef3ea768f71dd9f: Neither blocks nor log segments found. Creating new log.
I20260925 04:57:40.431402 19677 log.cc:824] T 3fbb415fd2514162b4af574453d8caf7 P d69cbdc154b54c568ef3ea768f71dd9f: Log is configured to *not* fsync() on all Append() calls
I20260925 04:57:40.433199 19679 tablet_bootstrap.cc:654] T 3fbb415fd2514162b4af574453d8caf7 P 9c30a7d195dc40c394ba71c26efe310a: Neither blocks nor log segments found. Creating new log.
I20260925 04:57:40.433286 19678 tablet_bootstrap.cc:654] T 3fbb415fd2514162b4af574453d8caf7 P c3fb4ba0ec384f07b318e85bb1ddf6ec: Neither blocks nor log segments found. Creating new log.
I20260925 04:57:40.434487 19679 log.cc:824] T 3fbb415fd2514162b4af574453d8caf7 P 9c30a7d195dc40c394ba71c26efe310a: Log is configured to *not* fsync() on all Append() calls
I20260925 04:57:40.434610 19678 log.cc:824] T 3fbb415fd2514162b4af574453d8caf7 P c3fb4ba0ec384f07b318e85bb1ddf6ec: Log is configured to *not* fsync() on all Append() calls
I20260925 04:57:40.435302 19677 tablet_bootstrap.cc:492] T 3fbb415fd2514162b4af574453d8caf7 P d69cbdc154b54c568ef3ea768f71dd9f: No bootstrap required, opened a new log
I20260925 04:57:40.435662 19677 ts_tablet_manager.cc:1397] T 3fbb415fd2514162b4af574453d8caf7 P d69cbdc154b54c568ef3ea768f71dd9f: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260925 04:57:40.437247 19679 tablet_bootstrap.cc:492] T 3fbb415fd2514162b4af574453d8caf7 P 9c30a7d195dc40c394ba71c26efe310a: No bootstrap required, opened a new log
I20260925 04:57:40.437539 19679 ts_tablet_manager.cc:1397] T 3fbb415fd2514162b4af574453d8caf7 P 9c30a7d195dc40c394ba71c26efe310a: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260925 04:57:40.438206 19678 tablet_bootstrap.cc:492] T 3fbb415fd2514162b4af574453d8caf7 P c3fb4ba0ec384f07b318e85bb1ddf6ec: No bootstrap required, opened a new log
I20260925 04:57:40.438500 19678 ts_tablet_manager.cc:1397] T 3fbb415fd2514162b4af574453d8caf7 P c3fb4ba0ec384f07b318e85bb1ddf6ec: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.002s
I20260925 04:57:40.445377 19679 raft_consensus.cc:348] T 3fbb415fd2514162b4af574453d8caf7 P 9c30a7d195dc40c394ba71c26efe310a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c30a7d195dc40c394ba71c26efe310a" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 38483 } } peers { permanent_uuid: "c3fb4ba0ec384f07b318e85bb1ddf6ec" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 36947 } } peers { permanent_uuid: "d69cbdc154b54c568ef3ea768f71dd9f" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 36597 } }
I20260925 04:57:40.445770 19679 raft_consensus.cc:374] T 3fbb415fd2514162b4af574453d8caf7 P 9c30a7d195dc40c394ba71c26efe310a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:57:40.445904 19679 raft_consensus.cc:729] T 3fbb415fd2514162b4af574453d8caf7 P 9c30a7d195dc40c394ba71c26efe310a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c30a7d195dc40c394ba71c26efe310a, State: Initialized, Role: FOLLOWER
I20260925 04:57:40.446552 19679 consensus_queue.cc:260] T 3fbb415fd2514162b4af574453d8caf7 P 9c30a7d195dc40c394ba71c26efe310a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c30a7d195dc40c394ba71c26efe310a" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 38483 } } peers { permanent_uuid: "c3fb4ba0ec384f07b318e85bb1ddf6ec" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 36947 } } peers { permanent_uuid: "d69cbdc154b54c568ef3ea768f71dd9f" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 36597 } }
I20260925 04:57:40.446614 19677 raft_consensus.cc:348] T 3fbb415fd2514162b4af574453d8caf7 P d69cbdc154b54c568ef3ea768f71dd9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c30a7d195dc40c394ba71c26efe310a" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 38483 } } peers { permanent_uuid: "c3fb4ba0ec384f07b318e85bb1ddf6ec" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 36947 } } peers { permanent_uuid: "d69cbdc154b54c568ef3ea768f71dd9f" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 36597 } }
I20260925 04:57:40.447119 19677 raft_consensus.cc:374] T 3fbb415fd2514162b4af574453d8caf7 P d69cbdc154b54c568ef3ea768f71dd9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:57:40.447255 19677 raft_consensus.cc:729] T 3fbb415fd2514162b4af574453d8caf7 P d69cbdc154b54c568ef3ea768f71dd9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d69cbdc154b54c568ef3ea768f71dd9f, State: Initialized, Role: FOLLOWER
I20260925 04:57:40.447685 19649 heartbeater.cc:503] Master 127.6.29.190:39733 was elected leader, sending a full tablet report...
I20260925 04:57:40.447965 19677 consensus_queue.cc:260] T 3fbb415fd2514162b4af574453d8caf7 P d69cbdc154b54c568ef3ea768f71dd9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c30a7d195dc40c394ba71c26efe310a" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 38483 } } peers { permanent_uuid: "c3fb4ba0ec384f07b318e85bb1ddf6ec" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 36947 } } peers { permanent_uuid: "d69cbdc154b54c568ef3ea768f71dd9f" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 36597 } }
I20260925 04:57:40.448468 19679 ts_tablet_manager.cc:1428] T 3fbb415fd2514162b4af574453d8caf7 P 9c30a7d195dc40c394ba71c26efe310a: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260925 04:57:40.449606 19677 ts_tablet_manager.cc:1428] T 3fbb415fd2514162b4af574453d8caf7 P d69cbdc154b54c568ef3ea768f71dd9f: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260925 04:57:40.450428 19678 raft_consensus.cc:348] T 3fbb415fd2514162b4af574453d8caf7 P c3fb4ba0ec384f07b318e85bb1ddf6ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c30a7d195dc40c394ba71c26efe310a" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 38483 } } peers { permanent_uuid: "c3fb4ba0ec384f07b318e85bb1ddf6ec" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 36947 } } peers { permanent_uuid: "d69cbdc154b54c568ef3ea768f71dd9f" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 36597 } }
I20260925 04:57:40.450793 19678 raft_consensus.cc:374] T 3fbb415fd2514162b4af574453d8caf7 P c3fb4ba0ec384f07b318e85bb1ddf6ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:57:40.450913 19678 raft_consensus.cc:729] T 3fbb415fd2514162b4af574453d8caf7 P c3fb4ba0ec384f07b318e85bb1ddf6ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3fb4ba0ec384f07b318e85bb1ddf6ec, State: Initialized, Role: FOLLOWER
I20260925 04:57:40.451476 19678 consensus_queue.cc:260] T 3fbb415fd2514162b4af574453d8caf7 P c3fb4ba0ec384f07b318e85bb1ddf6ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c30a7d195dc40c394ba71c26efe310a" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 38483 } } peers { permanent_uuid: "c3fb4ba0ec384f07b318e85bb1ddf6ec" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 36947 } } peers { permanent_uuid: "d69cbdc154b54c568ef3ea768f71dd9f" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 36597 } }
I20260925 04:57:40.453187 19370 heartbeater.cc:503] Master 127.6.29.190:39733 was elected leader, sending a full tablet report...
I20260925 04:57:40.455128 19678 ts_tablet_manager.cc:1428] T 3fbb415fd2514162b4af574453d8caf7 P c3fb4ba0ec384f07b318e85bb1ddf6ec: Time spent starting tablet: real 0.017s user 0.012s sys 0.002s
W20260925 04:57:40.565986 19652 tablet.cc:2369] T 3fbb415fd2514162b4af574453d8caf7 P 9c30a7d195dc40c394ba71c26efe310a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:57:40.574185 19683 raft_consensus.cc:482] T 3fbb415fd2514162b4af574453d8caf7 P 9c30a7d195dc40c394ba71c26efe310a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:57:40.574457 19683 raft_consensus.cc:504] T 3fbb415fd2514162b4af574453d8caf7 P 9c30a7d195dc40c394ba71c26efe310a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c30a7d195dc40c394ba71c26efe310a" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 38483 } } peers { permanent_uuid: "c3fb4ba0ec384f07b318e85bb1ddf6ec" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 36947 } } peers { permanent_uuid: "d69cbdc154b54c568ef3ea768f71dd9f" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 36597 } }
I20260925 04:57:40.576237 19683 leader_election.cc:288] T 3fbb415fd2514162b4af574453d8caf7 P 9c30a7d195dc40c394ba71c26efe310a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3fb4ba0ec384f07b318e85bb1ddf6ec (127.6.29.129:36947), d69cbdc154b54c568ef3ea768f71dd9f (127.6.29.130:36597)
I20260925 04:57:40.582535 19674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:40.576572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:48381 (local address 127.6.29.129:36947)
0925 04:57:40.576762 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:40.576770 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:40.577067 (+ 297us) server_negotiation.cc:407] Connection header received
0925 04:57:40.577154 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:40.577159 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:40.577234 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:40.577330 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:57:40.577860 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:40.578825 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:40.580683 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:40.581666 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:40.581789 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:40.581843 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:40.582184 (+ 341us) server_negotiation.cc:299] Negotiation successful
0925 04:57:40.582274 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 04:57:40.582654 19653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:40.576497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:36947 (local address 127.6.29.131:48381)
0925 04:57:40.576760 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:40.576780 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 04:57:40.576930 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:57:40.577372 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:57:40.577376 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:57:40.577401 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:57:40.577723 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:40.577736 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:40.578966 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:40.578971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:40.580525 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:40.580537 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:40.581825 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:40.581831 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:40.582017 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:40.582053 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 04:57:40.582156 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 04:57:40.582291 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":3,"spinlock_wait_cycles":21888}
I20260925 04:57:40.583302 19686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:40.576553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:36597 (local address 127.6.29.131:48081)
0925 04:57:40.577344 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:40.577362 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:57:40.577470 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:57:40.577857 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:57:40.577861 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:57:40.577909 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:57:40.578306 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:40.578318 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:40.579475 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:40.579478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:40.581097 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:40.581110 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:40.582547 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:40.582555 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:40.582741 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:40.582773 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 04:57:40.582875 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 04:57:40.583013 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":421,"threads_started":1}
I20260925 04:57:40.583302 19675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:40.576792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:48081 (local address 127.6.29.130:36597)
0925 04:57:40.577042 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:40.577049 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:40.577449 (+ 400us) server_negotiation.cc:407] Connection header received
0925 04:57:40.577610 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:40.577616 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:40.577729 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:40.577831 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:57:40.578464 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:40.579316 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:40.581260 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:40.582334 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:40.582494 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:40.582561 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:40.582914 (+ 353us) server_negotiation.cc:299] Negotiation successful
0925 04:57:40.583047 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 04:57:40.583799 19325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fbb415fd2514162b4af574453d8caf7" candidate_uuid: "9c30a7d195dc40c394ba71c26efe310a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3fb4ba0ec384f07b318e85bb1ddf6ec" is_pre_election: true
I20260925 04:57:40.584528 19325 raft_consensus.cc:2454] T 3fbb415fd2514162b4af574453d8caf7 P c3fb4ba0ec384f07b318e85bb1ddf6ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c30a7d195dc40c394ba71c26efe310a in term 0.
I20260925 04:57:40.584408 19469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fbb415fd2514162b4af574453d8caf7" candidate_uuid: "9c30a7d195dc40c394ba71c26efe310a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d69cbdc154b54c568ef3ea768f71dd9f" is_pre_election: true
I20260925 04:57:40.585000 19469 raft_consensus.cc:2454] T 3fbb415fd2514162b4af574453d8caf7 P d69cbdc154b54c568ef3ea768f71dd9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c30a7d195dc40c394ba71c26efe310a in term 0.
I20260925 04:57:40.585341 19538 leader_election.cc:302] T 3fbb415fd2514162b4af574453d8caf7 P 9c30a7d195dc40c394ba71c26efe310a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c30a7d195dc40c394ba71c26efe310a, c3fb4ba0ec384f07b318e85bb1ddf6ec; no voters:
I20260925 04:57:40.585779 19683 raft_consensus.cc:2781] T 3fbb415fd2514162b4af574453d8caf7 P 9c30a7d195dc40c394ba71c26efe310a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:57:40.585927 19683 raft_consensus.cc:482] T 3fbb415fd2514162b4af574453d8caf7 P 9c30a7d195dc40c394ba71c26efe310a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:57:40.586035 19683 raft_consensus.cc:3037] T 3fbb415fd2514162b4af574453d8caf7 P 9c30a7d195dc40c394ba71c26efe310a [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:57:40.588275 19683 raft_consensus.cc:504] T 3fbb415fd2514162b4af574453d8caf7 P 9c30a7d195dc40c394ba71c26efe310a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c30a7d195dc40c394ba71c26efe310a" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 38483 } } peers { permanent_uuid: "c3fb4ba0ec384f07b318e85bb1ddf6ec" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 36947 } } peers { permanent_uuid: "d69cbdc154b54c568ef3ea768f71dd9f" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 36597 } }
I20260925 04:57:40.588914 19683 leader_election.cc:288] T 3fbb415fd2514162b4af574453d8caf7 P 9c30a7d195dc40c394ba71c26efe310a [CANDIDATE]: Term 1 election: Requested vote from peers c3fb4ba0ec384f07b318e85bb1ddf6ec (127.6.29.129:36947), d69cbdc154b54c568ef3ea768f71dd9f (127.6.29.130:36597)
I20260925 04:57:40.589390 19325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fbb415fd2514162b4af574453d8caf7" candidate_uuid: "9c30a7d195dc40c394ba71c26efe310a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3fb4ba0ec384f07b318e85bb1ddf6ec"
I20260925 04:57:40.589491 19469 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fbb415fd2514162b4af574453d8caf7" candidate_uuid: "9c30a7d195dc40c394ba71c26efe310a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d69cbdc154b54c568ef3ea768f71dd9f"
I20260925 04:57:40.589603 19325 raft_consensus.cc:3037] T 3fbb415fd2514162b4af574453d8caf7 P c3fb4ba0ec384f07b318e85bb1ddf6ec [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:57:40.589744 19469 raft_consensus.cc:3037] T 3fbb415fd2514162b4af574453d8caf7 P d69cbdc154b54c568ef3ea768f71dd9f [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:57:40.591904 19325 raft_consensus.cc:2454] T 3fbb415fd2514162b4af574453d8caf7 P c3fb4ba0ec384f07b318e85bb1ddf6ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c30a7d195dc40c394ba71c26efe310a in term 1.
I20260925 04:57:40.592211 19469 raft_consensus.cc:2454] T 3fbb415fd2514162b4af574453d8caf7 P d69cbdc154b54c568ef3ea768f71dd9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c30a7d195dc40c394ba71c26efe310a in term 1.
I20260925 04:57:40.592264 19538 leader_election.cc:302] T 3fbb415fd2514162b4af574453d8caf7 P 9c30a7d195dc40c394ba71c26efe310a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c30a7d195dc40c394ba71c26efe310a, c3fb4ba0ec384f07b318e85bb1ddf6ec; no voters:
I20260925 04:57:40.592633 19683 raft_consensus.cc:2781] T 3fbb415fd2514162b4af574453d8caf7 P 9c30a7d195dc40c394ba71c26efe310a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:57:40.593283 19683 raft_consensus.cc:686] T 3fbb415fd2514162b4af574453d8caf7 P 9c30a7d195dc40c394ba71c26efe310a [term 1 LEADER]: Becoming Leader. State: Replica: 9c30a7d195dc40c394ba71c26efe310a, State: Running, Role: LEADER
I20260925 04:57:40.593755 19683 consensus_queue.cc:237] T 3fbb415fd2514162b4af574453d8caf7 P 9c30a7d195dc40c394ba71c26efe310a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c30a7d195dc40c394ba71c26efe310a" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 38483 } } peers { permanent_uuid: "c3fb4ba0ec384f07b318e85bb1ddf6ec" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 36947 } } peers { permanent_uuid: "d69cbdc154b54c568ef3ea768f71dd9f" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 36597 } }
I20260925 04:57:40.598482 19172 catalog_manager.cc:5495] T 3fbb415fd2514162b4af574453d8caf7 P 9c30a7d195dc40c394ba71c26efe310a reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c30a7d195dc40c394ba71c26efe310a (127.6.29.131). New cstate: current_term: 1 leader_uuid: "9c30a7d195dc40c394ba71c26efe310a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c30a7d195dc40c394ba71c26efe310a" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 38483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3fb4ba0ec384f07b318e85bb1ddf6ec" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 36947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d69cbdc154b54c568ef3ea768f71dd9f" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 36597 } health_report { overall_health: UNKNOWN } } }
W20260925 04:57:40.606705 19516 tablet.cc:2369] T 3fbb415fd2514162b4af574453d8caf7 P d69cbdc154b54c568ef3ea768f71dd9f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:57:40.644627 19673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:40.638797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42480 (local address 127.6.29.131:38483)
0925 04:57:40.638979 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:40.638984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:40.639102 (+ 118us) server_negotiation.cc:407] Connection header received
0925 04:57:40.639240 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:40.639244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:40.639326 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:40.639571 (+ 245us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:57:40.639964 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:40.640808 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:40.642391 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:40.642814 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:40.642919 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:40.643447 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:57:40.644022 (+ 575us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:57:40.644192 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:40.644274 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 04:57:40.644357 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 04:57:40.652858 19325 raft_consensus.cc:1261] T 3fbb415fd2514162b4af574453d8caf7 P c3fb4ba0ec384f07b318e85bb1ddf6ec [term 1 FOLLOWER]: Refusing update from remote peer 9c30a7d195dc40c394ba71c26efe310a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:57:40.652864 19469 raft_consensus.cc:1261] T 3fbb415fd2514162b4af574453d8caf7 P d69cbdc154b54c568ef3ea768f71dd9f [term 1 FOLLOWER]: Refusing update from remote peer 9c30a7d195dc40c394ba71c26efe310a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:57:40.653838 19687 consensus_queue.cc:1035] T 3fbb415fd2514162b4af574453d8caf7 P 9c30a7d195dc40c394ba71c26efe310a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3fb4ba0ec384f07b318e85bb1ddf6ec" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 36947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:57:40.654186 19683 consensus_queue.cc:1035] T 3fbb415fd2514162b4af574453d8caf7 P 9c30a7d195dc40c394ba71c26efe310a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d69cbdc154b54c568ef3ea768f71dd9f" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 36597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:57:40.680150 19692 mvcc.cc:205] Tried to move back new op lower bound from 7333119019623940096 to 7333119019396517888. Current Snapshot: MvccSnapshot[applied={T|T < 7333119019623940096}]
I20260925 04:57:40.689730 19173 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:57:40.690055 19173 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:57:40.694109 19173 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 87f5df83da2248e49d76f921eb4e52ff: Sending DeleteTablet for 3 replicas of tablet 3fbb415fd2514162b4af574453d8caf7
I20260925 04:57:40.695362 19305 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fbb415fd2514162b4af574453d8caf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:57:40 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:53818
I20260925 04:57:40.695500 19449 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fbb415fd2514162b4af574453d8caf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:57:40 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:42030
I20260925 04:57:40.695976 19584 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fbb415fd2514162b4af574453d8caf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:57:40 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:42466
I20260925 04:57:40.704387 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 19210
I20260925 04:57:40.706694 19701 tablet_replica.cc:330] T 3fbb415fd2514162b4af574453d8caf7 P c3fb4ba0ec384f07b318e85bb1ddf6ec: stopping tablet replica
I20260925 04:57:40.706678 19703 tablet_replica.cc:330] T 3fbb415fd2514162b4af574453d8caf7 P 9c30a7d195dc40c394ba71c26efe310a: stopping tablet replica
I20260925 04:57:40.707198 19701 raft_consensus.cc:2229] T 3fbb415fd2514162b4af574453d8caf7 P c3fb4ba0ec384f07b318e85bb1ddf6ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:57:40.707208 19703 raft_consensus.cc:2229] T 3fbb415fd2514162b4af574453d8caf7 P 9c30a7d195dc40c394ba71c26efe310a [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:57:40.707881 19701 raft_consensus.cc:2258] T 3fbb415fd2514162b4af574453d8caf7 P c3fb4ba0ec384f07b318e85bb1ddf6ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:57:40.707967 19703 raft_consensus.cc:2258] T 3fbb415fd2514162b4af574453d8caf7 P 9c30a7d195dc40c394ba71c26efe310a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:57:40.710314 19701 ts_tablet_manager.cc:1905] T 3fbb415fd2514162b4af574453d8caf7 P c3fb4ba0ec384f07b318e85bb1ddf6ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:57:40.710320 19703 ts_tablet_manager.cc:1905] T 3fbb415fd2514162b4af574453d8caf7 P 9c30a7d195dc40c394ba71c26efe310a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:57:40.714390 19702 tablet_replica.cc:330] T 3fbb415fd2514162b4af574453d8caf7 P d69cbdc154b54c568ef3ea768f71dd9f: stopping tablet replica
I20260925 04:57:40.714965 19702 raft_consensus.cc:2229] T 3fbb415fd2514162b4af574453d8caf7 P d69cbdc154b54c568ef3ea768f71dd9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:57:40.715582 19702 raft_consensus.cc:2258] T 3fbb415fd2514162b4af574453d8caf7 P d69cbdc154b54c568ef3ea768f71dd9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:57:40.718317 19701 ts_tablet_manager.cc:1918] T 3fbb415fd2514162b4af574453d8caf7 P c3fb4ba0ec384f07b318e85bb1ddf6ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:57:40.718544 19701 log.cc:1196] T 3fbb415fd2514162b4af574453d8caf7 P c3fb4ba0ec384f07b318e85bb1ddf6ec: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-0/wal/wals/3fbb415fd2514162b4af574453d8caf7
I20260925 04:57:40.719213 19701 ts_tablet_manager.cc:1939] T 3fbb415fd2514162b4af574453d8caf7 P c3fb4ba0ec384f07b318e85bb1ddf6ec: Deleting consensus metadata
I20260925 04:57:40.720572 19702 ts_tablet_manager.cc:1905] T 3fbb415fd2514162b4af574453d8caf7 P d69cbdc154b54c568ef3ea768f71dd9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:57:40.721415 19159 catalog_manager.cc:4841] TS c3fb4ba0ec384f07b318e85bb1ddf6ec (127.6.29.129:36947): tablet 3fbb415fd2514162b4af574453d8caf7 (table test-table [id=c99652aef6114b6b9a14d051f0101dae]) successfully deleted
I20260925 04:57:40.726692 19702 ts_tablet_manager.cc:1918] T 3fbb415fd2514162b4af574453d8caf7 P d69cbdc154b54c568ef3ea768f71dd9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:57:40.726879 19702 log.cc:1196] T 3fbb415fd2514162b4af574453d8caf7 P d69cbdc154b54c568ef3ea768f71dd9f: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-1/wal/wals/3fbb415fd2514162b4af574453d8caf7
I20260925 04:57:40.727376 19702 ts_tablet_manager.cc:1939] T 3fbb415fd2514162b4af574453d8caf7 P d69cbdc154b54c568ef3ea768f71dd9f: Deleting consensus metadata
I20260925 04:57:40.727833 19703 ts_tablet_manager.cc:1918] T 3fbb415fd2514162b4af574453d8caf7 P 9c30a7d195dc40c394ba71c26efe310a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:57:40.728039 19703 log.cc:1196] T 3fbb415fd2514162b4af574453d8caf7 P 9c30a7d195dc40c394ba71c26efe310a: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790312053666503-6262-0/minicluster-data/ts-2/wal/wals/3fbb415fd2514162b4af574453d8caf7
I20260925 04:57:40.728571 19703 ts_tablet_manager.cc:1939] T 3fbb415fd2514162b4af574453d8caf7 P 9c30a7d195dc40c394ba71c26efe310a: Deleting consensus metadata
I20260925 04:57:40.729454 19157 catalog_manager.cc:4841] TS d69cbdc154b54c568ef3ea768f71dd9f (127.6.29.130:36597): tablet 3fbb415fd2514162b4af574453d8caf7 (table test-table [id=c99652aef6114b6b9a14d051f0101dae]) successfully deleted
I20260925 04:57:40.730144 19157 catalog_manager.cc:4841] TS 9c30a7d195dc40c394ba71c26efe310a (127.6.29.131:38483): tablet 3fbb415fd2514162b4af574453d8caf7 (table test-table [id=c99652aef6114b6b9a14d051f0101dae]) successfully deleted
I20260925 04:57:40.732111 19674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:40.706460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53820 (local address 127.6.29.129:36947)
0925 04:57:40.706656 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:40.706663 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:40.706687 (+ 24us) server_negotiation.cc:407] Connection header received
0925 04:57:40.706758 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:40.706765 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:40.706876 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:40.706994 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:40.710785 (+ 3791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:40.712086 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:40.714173 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:40.714859 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:40.715010 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:40.719232 (+ 4222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:40.719249 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:40.719267 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:40.719315 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:40.724679 (+ 5364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:40.725630 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:40.725641 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:40.725646 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:40.725816 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:40.730391 (+ 4575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:40.730397 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:40.730401 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:40.730908 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:40.731232 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:40.731449 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 04:57:40.731697 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 04:57:40.732779 19365 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:57:40.826265 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 19210
I20260925 04:57:40.843918 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 19374
I20260925 04:57:40.860021 19675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:40.844880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42044 (local address 127.6.29.130:36597)
0925 04:57:40.845049 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:40.845053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:40.845072 (+ 19us) server_negotiation.cc:407] Connection header received
0925 04:57:40.845249 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:40.845254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:40.845318 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:40.845401 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:40.847156 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:40.848106 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:40.850003 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:40.850414 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:40.850521 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:40.853838 (+ 3317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:40.853853 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:40.853869 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:40.853910 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:40.857038 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:40.857898 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:40.857909 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:40.857914 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:40.858117 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:40.858544 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:40.858548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:40.858550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:40.858876 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:40.859129 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:40.859457 (+ 328us) server_negotiation.cc:299] Negotiation successful
0925 04:57:40.859621 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 04:57:40.860780 19510 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:57:40.962545 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 19374
I20260925 04:57:40.979910 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 19519
I20260925 04:57:40.997582 19673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:40.980718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42482 (local address 127.6.29.131:38483)
0925 04:57:40.980911 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:40.980921 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:40.981008 (+ 87us) server_negotiation.cc:407] Connection header received
0925 04:57:40.981298 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:40.981304 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:40.981396 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:40.981532 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:40.983483 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:40.984427 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:40.986347 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:40.986775 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:40.986878 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:40.990584 (+ 3706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:40.990598 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:40.990619 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:40.990661 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:40.994480 (+ 3819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:40.995442 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:40.995452 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:40.995461 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:40.995614 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:40.996109 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:40.996113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:40.996115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:40.996473 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:40.996788 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:40.997080 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 04:57:40.997242 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 04:57:40.998306 19644 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:57:41.123710 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 19519
I20260925 04:57:41.143515 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 19140
I20260925 04:57:41.144910 19203 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:57:41.258746 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 19140
I20260925 04:57:41.280416 6262 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 04:57:41.626179 6262 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 04:57:41.626407 6262 mini_ranger.cc:66] Stopping Ranger...
I20260925 04:57:41.972178 6262 mini_ranger.cc:68] Stopped Ranger
2026-09-25 04:57:41.972 UTC [18192] LOG: received smart shutdown request
2026-09-25 04:57:41.975 UTC [18192] LOG: background worker "logical replication launcher" (PID 18201) exited with exit code 1
2026-09-25 04:57:41.975 UTC [18196] LOG: shutting down
2026-09-25 04:57:42.164 UTC [18192] LOG: database system is shut down
2026-09-25T04:57:42Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52774 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:57:42 dist-test-slave-xzjr krb5kdc[19716](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:57:42 dist-test-slave-xzjr krb5kdc[19716](info): set up 2 sockets
Sep 25 04:57:42 dist-test-slave-xzjr krb5kdc[19716](info): commencing operation
krb5kdc: starting...
W20260925 04:57:43.408561 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 04:57:43 dist-test-slave-xzjr krb5kdc[19716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:57:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:57:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.29.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
I20260925 04:57:43.644752 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:40925
--webserver_interface=127.6.29.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:40925,127.6.29.189:40393,127.6.29.188:43147
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:57:43.979001 19729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:57:43.979342 19729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:57:43.979521 19729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:57:43.987323 19729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:57:43.987448 19729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:57:43.987547 19729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:57:43.987648 19729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:57:43.987733 19729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:57:43.997933 19729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:40925,127.6.29.189:40393,127.6.29.188:43147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:40925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:57:43.999328 19729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:57:44.000774 19729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:57:44.012722 19734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:44.012833 19737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:44.012851 19735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:57:44.013617 19729 server_base.cc:1034] running on GCE node
I20260925 04:57:44.014420 19729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:57:44.015976 19729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:57:44.017196 19729 hybrid_clock.cc:648] HybridClock initialized: now 1790312264017154 us; error 35 us; skew 500 ppm
I20260925 04:57:44.017586 19729 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:57:44 dist-test-slave-xzjr krb5kdc[19716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312264, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:57:44.026521 19729 init.cc:377] Logged in from keytab as kudu/127.6.29.190@KRBTEST.COM (short username kudu)
I20260925 04:57:44.028254 19729 webserver.cc:460] Webserver started at http://127.6.29.190:37213/ using document root <none> and password file <none>
I20260925 04:57:44.029014 19729 fs_manager.cc:359] Metadata directory not provided
I20260925 04:57:44.029147 19729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:57:44.029486 19729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:57:44.032621 19729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "f4c7addf0ebd4999826812bfa39e2c21"
format_stamp: "Formatted at 2026-09-25 04:57:44 on dist-test-slave-xzjr"
server_key: "0746cb8e1bb938da80750f6bb34eb765"
server_key_iv: "48617f9afe5d486a1b3b9f63bf28a254"
server_key_version: "encryptionkey@0"
I20260925 04:57:44.033440 19729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "f4c7addf0ebd4999826812bfa39e2c21"
format_stamp: "Formatted at 2026-09-25 04:57:44 on dist-test-slave-xzjr"
server_key: "0746cb8e1bb938da80750f6bb34eb765"
server_key_iv: "48617f9afe5d486a1b3b9f63bf28a254"
server_key_version: "encryptionkey@0"
I20260925 04:57:44.038777 19729 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 04:57:44.042342 19744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:44.043803 19729 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:57:44.044005 19729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "f4c7addf0ebd4999826812bfa39e2c21"
format_stamp: "Formatted at 2026-09-25 04:57:44 on dist-test-slave-xzjr"
server_key: "0746cb8e1bb938da80750f6bb34eb765"
server_key_iv: "48617f9afe5d486a1b3b9f63bf28a254"
server_key_version: "encryptionkey@0"
I20260925 04:57:44.044205 19729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:57:44.056944 19729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:57:44.060930 19729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:57:44.061242 19729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:57:44.080345 19729 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:40925
I20260925 04:57:44.080379 19795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:40925 every 8 connection(s)
I20260925 04:57:44.081749 19729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:57:44.085453 19796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:57:44.087749 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 19729
I20260925 04:57:44.088030 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 04:57:44.088683 6262 external_mini_cluster.cc:1428] Setting key 2d6ce1a4319312f0aa5f254199649d4f
I20260925 04:57:44.092144 19796 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40925 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 04:57:44 dist-test-slave-xzjr krb5kdc[19716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312264, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.29.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.189@KRBTEST.COM" created.
I20260925 04:57:44.117398 19800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:44.096596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42354 (local address 127.6.29.190:40925)
0925 04:57:44.097306 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:44.097325 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:44.097381 (+ 56us) server_negotiation.cc:407] Connection header received
0925 04:57:44.098707 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:44.098717 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:44.099016 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:44.099264 (+ 248us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:44.100518 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:44.102583 (+ 2065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:44.104138 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:44.105189 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:44.105426 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:44.109132 (+ 3706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:44.109147 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:44.109169 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:44.109214 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:44.112840 (+ 3626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:44.113815 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:44.113822 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:44.113825 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:44.113975 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:44.114485 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:44.114489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:44.114491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:44.115042 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:44.115202 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:44.116024 (+ 822us) server_negotiation.cc:299] Negotiation successful
0925 04:57:44.116277 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":310,"threads_started":1}
I20260925 04:57:44.117434 19799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:44.095256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:40925 (local address 127.0.0.1:42354)
0925 04:57:44.096119 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:44.096177 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 04:57:44.098271 (+ 2094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:57:44.099275 (+ 1004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:57:44.099280 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:57:44.099952 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:57:44.100302 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:44.100320 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:44.102781 (+ 2461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:44.102785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:44.103991 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:44.104004 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:44.105638 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:44.105646 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:44.105776 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:44.106522 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:57:44.106552 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:57:44.108845 (+ 2293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:57:44.112986 (+ 4141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:44.113001 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:44.113018 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:44.113645 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:44.114094 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:44.114102 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:44.114109 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:44.114329 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:44.115196 (+ 867us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:57:44.115208 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:57:44.115494 (+ 286us) client_negotiation.cc:769] Sending connection context
0925 04:57:44.115806 (+ 312us) client_negotiation.cc:240] Negotiation successful
0925 04:57:44.116521 (+ 715us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"mutex_wait_us":121,"thread_start_us":382,"threads_started":1}
I20260925 04:57:44.120864 19796 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.29.189" port: 40393 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:57:44.121997 19799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:44.121592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:40393 (local address 127.0.0.1:35908)
0925 04:57:44.121761 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:44.121878 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.189:40393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260925 04:57:44.122311 19745 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.29.189:40393: connect: Connection refused (error 111)
I20260925 04:57:44.124025 19799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:44.123607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:40393 (local address 127.0.0.1:35914)
0925 04:57:44.123791 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:44.123881 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.189:40393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 04:57:44.124253 19796 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.29.189:40393: Network error: Client connection negotiation failed: client connection to 127.6.29.189:40393: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.189.keytab.
Entry for principal kudu/127.6.29.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.189.keytab.
I20260925 04:57:44.143103 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.189:40393
--webserver_interface=127.6.29.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.189.keytab
--principal=kudu/127.6.29.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:40925,127.6.29.189:40393,127.6.29.188:43147
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:57:44.173599 19796 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.29.189" port: 40393 } attempt: 1
I20260925 04:57:44.174867 19799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:44.174324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:40393 (local address 127.0.0.1:35924)
0925 04:57:44.174564 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:44.174708 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.189:40393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 04:57:44.175974 19799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:44.175530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:40393 (local address 127.0.0.1:35936)
0925 04:57:44.175675 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:44.175758 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.189:40393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 04:57:44.176162 19796 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.29.189:40393: Network error: Client connection negotiation failed: client connection to 127.6.29.189:40393: connect: Connection refused (error 111)
I20260925 04:57:44.244498 19796 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.29.189" port: 40393 } attempt: 2
I20260925 04:57:44.245795 19799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:44.245221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:40393 (local address 127.0.0.1:35938)
0925 04:57:44.245492 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:44.245607 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.189:40393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 04:57:44.246872 19799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:44.246454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:40393 (local address 127.0.0.1:35946)
0925 04:57:44.246620 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:44.246716 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.189:40393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260925 04:57:44.247083 19796 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.29.189:40393: Network error: Client connection negotiation failed: client connection to 127.6.29.189:40393: connect: Connection refused (error 111)
I20260925 04:57:44.338374 19796 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.29.189" port: 40393 } attempt: 3
I20260925 04:57:44.339864 19799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:44.339190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:40393 (local address 127.0.0.1:35958)
0925 04:57:44.339470 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:44.339626 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.189:40393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260925 04:57:44.341102 19799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:44.340650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:40393 (local address 127.0.0.1:35964)
0925 04:57:44.340811 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:44.340897 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.189:40393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 04:57:44.341307 19796 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.29.189:40393: Network error: Client connection negotiation failed: client connection to 127.6.29.189:40393: connect: Connection refused (error 111)
W20260925 04:57:44.477298 19803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:57:44.477597 19803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:57:44.477690 19803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 04:57:44.484586 19796 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.29.189" port: 40393 } attempt: 4
W20260925 04:57:44.485093 19803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:57:44.485186 19803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:57:44.485231 19803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:57:44.485281 19803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:57:44.485355 19803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:57:44.486090 19799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:44.485494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:40393 (local address 127.0.0.1:35978)
0925 04:57:44.485818 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:44.485928 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.189:40393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
I20260925 04:57:44.487368 19799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:44.486935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:40393 (local address 127.0.0.1:35992)
0925 04:57:44.487091 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:44.487203 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.189:40393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 04:57:44.487610 19796 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.29.189:40393: Network error: Client connection negotiation failed: client connection to 127.6.29.189:40393: connect: Connection refused (error 111)
I20260925 04:57:44.496208 19803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:40925,127.6.29.189:40393,127.6.29.188:43147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.189.keytab
--principal=kudu/127.6.29.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.189:40393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.29.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:57:44.497593 19803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:57:44.498942 19803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:57:44.511698 19808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:44.511631 19809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:44.511619 19811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:57:44.512512 19803 server_base.cc:1034] running on GCE node
I20260925 04:57:44.513367 19803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:57:44.514923 19803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:57:44.516201 19803 hybrid_clock.cc:648] HybridClock initialized: now 1790312264516152 us; error 30 us; skew 500 ppm
I20260925 04:57:44.516556 19803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:57:44 dist-test-slave-xzjr krb5kdc[19716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312264, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:57:44.520028 19803 init.cc:377] Logged in from keytab as kudu/127.6.29.189@KRBTEST.COM (short username kudu)
I20260925 04:57:44.521690 19803 webserver.cc:460] Webserver started at http://127.6.29.189:34201/ using document root <none> and password file <none>
I20260925 04:57:44.522454 19803 fs_manager.cc:359] Metadata directory not provided
I20260925 04:57:44.522604 19803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:57:44.522935 19803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:57:44.526155 19803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-1/data/instance:
uuid: "31cc47688e7943048a38fa3aed8cc54a"
format_stamp: "Formatted at 2026-09-25 04:57:44 on dist-test-slave-xzjr"
server_key: "0e5a7f77c95c541a7f674be093294df9"
server_key_iv: "3626e5d5b4e35a69948f10869498ffc3"
server_key_version: "encryptionkey@0"
I20260925 04:57:44.527002 19803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-1/wal/instance:
uuid: "31cc47688e7943048a38fa3aed8cc54a"
format_stamp: "Formatted at 2026-09-25 04:57:44 on dist-test-slave-xzjr"
server_key: "0e5a7f77c95c541a7f674be093294df9"
server_key_iv: "3626e5d5b4e35a69948f10869498ffc3"
server_key_version: "encryptionkey@0"
I20260925 04:57:44.532702 19803 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 04:57:44.536520 19818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:44.538105 19803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:57:44.538336 19803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-1/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-1/wal
uuid: "31cc47688e7943048a38fa3aed8cc54a"
format_stamp: "Formatted at 2026-09-25 04:57:44 on dist-test-slave-xzjr"
server_key: "0e5a7f77c95c541a7f674be093294df9"
server_key_iv: "3626e5d5b4e35a69948f10869498ffc3"
server_key_version: "encryptionkey@0"
I20260925 04:57:44.538597 19803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:57:44.553053 19803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:57:44.557662 19803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:57:44.557996 19803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:57:44.577570 19803 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.189:40393
I20260925 04:57:44.577616 19869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.189:40393 every 8 connection(s)
I20260925 04:57:44.579083 19803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-1/data/info.pb
I20260925 04:57:44.583051 19870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:57:44.585568 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 19803
I20260925 04:57:44.585786 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-1/wal/instance
I20260925 04:57:44.586267 6262 external_mini_cluster.cc:1428] Setting key 2470555de3767e30554d61cab90367d3
I20260925 04:57:44.589983 19870 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40925 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 04:57:44 dist-test-slave-xzjr krb5kdc[19716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312264, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.189@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.29.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.188@KRBTEST.COM" created.
I20260925 04:57:44.612983 19800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:44.592885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42368 (local address 127.6.29.190:40925)
0925 04:57:44.593115 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:44.593119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:44.594685 (+ 1566us) server_negotiation.cc:407] Connection header received
0925 04:57:44.596698 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:44.596706 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:44.596810 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:44.596934 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:44.598577 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:44.599690 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:44.601449 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:44.602224 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:44.602379 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:44.606355 (+ 3976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:44.606363 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:44.606369 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:44.606416 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:44.609502 (+ 3086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:44.610523 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:44.610527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:44.610529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:44.610654 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:44.611167 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:44.611169 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:44.611171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:44.611462 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:44.611573 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:44.612554 (+ 981us) server_negotiation.cc:299] Negotiation successful
0925 04:57:44.612742 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260925 04:57:44.614638 19873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:44.593567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:40925 (local address 127.0.0.1:42368)
0925 04:57:44.594478 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:44.594531 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 04:57:44.596394 (+ 1863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:57:44.597141 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:57:44.597149 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:57:44.597743 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:57:44.598363 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:44.598382 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:44.599934 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:44.599939 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:44.601262 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:44.601292 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:44.602501 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:44.602516 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:44.602736 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:44.603793 (+ 1057us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:57:44.603835 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:57:44.606082 (+ 2247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:57:44.609668 (+ 3586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:44.609678 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:44.609697 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:44.610390 (+ 693us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:44.610771 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:44.610779 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:44.610787 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:44.611050 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:44.611616 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:57:44.611627 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:57:44.612141 (+ 514us) client_negotiation.cc:769] Sending connection context
0925 04:57:44.612469 (+ 328us) client_negotiation.cc:240] Negotiation successful
0925 04:57:44.613056 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":397,"threads_started":1}
I20260925 04:57:44.616542 19870 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.29.189" port: 40393 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 04:57:44 dist-test-slave-xzjr krb5kdc[19716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312264, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.189@KRBTEST.COM for kudu/127.6.29.189@KRBTEST.COM
I20260925 04:57:44.634307 19873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:44.617350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:40393 (local address 127.0.0.1:36006)
0925 04:57:44.617564 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:44.617587 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 04:57:44.617712 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:57:44.618819 (+ 1107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:57:44.618822 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:57:44.619172 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:57:44.619316 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:44.619323 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:44.621426 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:44.621430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:44.622261 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:44.622271 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:44.623573 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:44.623578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:44.623748 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:44.624518 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:57:44.624535 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:57:44.626820 (+ 2285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:57:44.631605 (+ 4785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:44.631609 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:44.631612 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:44.632086 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:44.632562 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:44.632566 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:44.632569 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:44.632719 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:44.633530 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:57:44.633534 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:57:44.633653 (+ 119us) client_negotiation.cc:769] Sending connection context
0925 04:57:44.633767 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 04:57:44.633996 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 04:57:44.634516 19875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:44.617641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36006 (local address 127.6.29.189:40393)
0925 04:57:44.618330 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:44.618343 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:44.618370 (+ 27us) server_negotiation.cc:407] Connection header received
0925 04:57:44.618417 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:44.618422 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:44.618653 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:44.618810 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:44.619477 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:44.621283 (+ 1806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:44.622411 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:44.623375 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:44.623527 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:44.627058 (+ 3531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:44.627076 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:44.627098 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:44.627141 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:44.631360 (+ 4219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:44.632233 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:44.632246 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:44.632251 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:44.632410 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:44.632846 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:44.632854 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:44.632859 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:44.633352 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:44.633592 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:44.633941 (+ 349us) server_negotiation.cc:299] Negotiation successful
0925 04:57:44.634140 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":403,"threads_started":1}
I20260925 04:57:44.637353 19870 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.29.188" port: 43147 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:57:44.638782 19873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:44.638218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:43147 (local address 127.0.0.1:51710)
0925 04:57:44.638482 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:44.638619 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:43147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260925 04:57:44.639176 19821 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.29.188:43147: connect: Connection refused (error 111)
I20260925 04:57:44.641006 19873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:44.640589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:43147 (local address 127.0.0.1:51712)
0925 04:57:44.640782 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:44.640883 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:43147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 04:57:44.641279 19870 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.29.188:43147: Network error: Client connection negotiation failed: client connection to 127.6.29.188:43147: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.188.keytab.
Entry for principal kudu/127.6.29.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.188.keytab.
I20260925 04:57:44.643616 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.188:43147
--webserver_interface=127.6.29.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.188.keytab
--principal=kudu/127.6.29.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:35351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:40925,127.6.29.189:40393,127.6.29.188:43147
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:57:44.690639 19870 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.29.188" port: 43147 } attempt: 1
I20260925 04:57:44.692099 19873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:44.691502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:43147 (local address 127.0.0.1:51724)
0925 04:57:44.691732 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:44.691923 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:43147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260925 04:57:44.693238 19873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:44.692854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:43147 (local address 127.0.0.1:51730)
0925 04:57:44.693036 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:44.693122 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:43147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 04:57:44.693460 19870 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.29.188:43147: Network error: Client connection negotiation failed: client connection to 127.6.29.188:43147: connect: Connection refused (error 111)
I20260925 04:57:44.761695 19870 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.29.188" port: 43147 } attempt: 2
I20260925 04:57:44.762841 19873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:44.762387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:43147 (local address 127.0.0.1:51734)
0925 04:57:44.762587 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:44.762687 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:43147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 04:57:44.763959 19873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:44.763569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:43147 (local address 127.0.0.1:51746)
0925 04:57:44.763724 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:44.763848 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:43147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 04:57:44.764178 19870 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.29.188:43147: Network error: Client connection negotiation failed: client connection to 127.6.29.188:43147: connect: Connection refused (error 111)
I20260925 04:57:44.786911 19796 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.29.189" port: 40393 } attempt: 5
Sep 25 04:57:44 dist-test-slave-xzjr krb5kdc[19716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312264, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for kudu/127.6.29.189@KRBTEST.COM
I20260925 04:57:44.802738 19799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:44.787626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:40393 (local address 127.0.0.1:36020)
0925 04:57:44.787888 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:44.787911 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 04:57:44.788056 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:57:44.788671 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:57:44.788675 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:57:44.789063 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:57:44.789240 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:44.789248 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:44.790682 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:44.790686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:44.791613 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:44.791625 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:44.792680 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:44.792685 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:44.792820 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:44.793502 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:57:44.793521 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:57:44.795983 (+ 2462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:57:44.800037 (+ 4054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:44.800041 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:44.800044 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:44.800463 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:44.800931 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:44.800935 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:44.800937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:44.801093 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:44.801731 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:57:44.801735 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:57:44.801900 (+ 165us) client_negotiation.cc:769] Sending connection context
0925 04:57:44.802052 (+ 152us) client_negotiation.cc:240] Negotiation successful
0925 04:57:44.802494 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 04:57:44.802764 19875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:44.787795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36020 (local address 127.6.29.189:40393)
0925 04:57:44.788306 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:44.788313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:44.788331 (+ 18us) server_negotiation.cc:407] Connection header received
0925 04:57:44.788396 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:44.788401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:44.788498 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:44.788615 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:44.789415 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:44.790527 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:44.791798 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:44.792518 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:44.792648 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:44.796218 (+ 3570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:44.796227 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:44.796233 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:44.796287 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:44.799870 (+ 3583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:44.800622 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:44.800628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:44.800632 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:44.800787 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:44.801205 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:44.801210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:44.801213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:44.801586 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:44.801737 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:44.802187 (+ 450us) server_negotiation.cc:299] Negotiation successful
0925 04:57:44.802439 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374}
I20260925 04:57:44.804150 19796 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.29.188" port: 43147 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:57:44.805284 19799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:44.804866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:43147 (local address 127.0.0.1:51758)
0925 04:57:44.805038 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:44.805141 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:43147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 04:57:44.806362 19799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:44.806029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:43147 (local address 127.0.0.1:51760)
0925 04:57:44.806189 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:44.806273 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:43147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260925 04:57:44.806602 19796 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.29.188:43147: Network error: Client connection negotiation failed: client connection to 127.6.29.188:43147: connect: Connection refused (error 111)
I20260925 04:57:44.855460 19870 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.29.188" port: 43147 } attempt: 3
I20260925 04:57:44.856700 19873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:44.856191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:43147 (local address 127.0.0.1:51776)
0925 04:57:44.856439 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:44.856539 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:43147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 04:57:44.857868 19873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:44.857430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:43147 (local address 127.0.0.1:51792)
0925 04:57:44.857662 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:44.857735 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:43147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260925 04:57:44.857848 19796 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.29.188" port: 43147 } attempt: 1
W20260925 04:57:44.858168 19870 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.29.188:43147: Network error: Client connection negotiation failed: client connection to 127.6.29.188:43147: connect: Connection refused (error 111)
I20260925 04:57:44.858911 19799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:44.858437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:43147 (local address 127.0.0.1:51802)
0925 04:57:44.858650 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:44.858747 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:43147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260925 04:57:44.859970 19799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:44.859541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:43147 (local address 127.0.0.1:51806)
0925 04:57:44.859704 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:44.859832 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:43147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 04:57:44.860204 19796 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.29.188:43147: Network error: Client connection negotiation failed: client connection to 127.6.29.188:43147: connect: Connection refused (error 111)
I20260925 04:57:44.928467 19796 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.29.188" port: 43147 } attempt: 2
I20260925 04:57:44.929896 19799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:44.929195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:43147 (local address 127.0.0.1:51816)
0925 04:57:44.929480 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:44.929614 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:43147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 04:57:44.931180 19799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:44.930722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:43147 (local address 127.0.0.1:51820)
0925 04:57:44.930916 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:44.931019 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:43147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260925 04:57:44.931457 19796 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.29.188:43147: Network error: Client connection negotiation failed: client connection to 127.6.29.188:43147: connect: Connection refused (error 111)
W20260925 04:57:44.991156 19877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:57:44.991474 19877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:57:44.991642 19877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:57:44.999143 19877 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:57:44.999260 19877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:57:44.999358 19877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:57:44.999449 19877 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:57:44.999554 19877 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:57:45.001437 19870 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.29.188" port: 43147 } attempt: 4
I20260925 04:57:45.002592 19873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:45.002110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:43147 (local address 127.0.0.1:51830)
0925 04:57:45.002360 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:45.002442 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:43147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 04:57:45.003703 19873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:45.003380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:43147 (local address 127.0.0.1:51836)
0925 04:57:45.003528 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:45.003600 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:43147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 04:57:45.004034 19870 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.29.188:43147: Network error: Client connection negotiation failed: client connection to 127.6.29.188:43147: connect: Connection refused (error 111)
I20260925 04:57:45.010478 19877 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:35351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:40925,127.6.29.189:40393,127.6.29.188:43147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.188.keytab
--principal=kudu/127.6.29.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.188:43147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.29.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:57:45.011952 19877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:57:45.013273 19877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:57:45.025642 19885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:45.025700 19883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:45.026279 19882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:57:45.026589 19877 server_base.cc:1034] running on GCE node
I20260925 04:57:45.027406 19877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:57:45.028966 19877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:57:45.030201 19877 hybrid_clock.cc:648] HybridClock initialized: now 1790312265030117 us; error 82 us; skew 500 ppm
I20260925 04:57:45.030578 19877 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:57:45 dist-test-slave-xzjr krb5kdc[19716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312265, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:57:45.033995 19877 init.cc:377] Logged in from keytab as kudu/127.6.29.188@KRBTEST.COM (short username kudu)
I20260925 04:57:45.035524 19877 webserver.cc:460] Webserver started at http://127.6.29.188:37809/ using document root <none> and password file <none>
I20260925 04:57:45.036296 19877 fs_manager.cc:359] Metadata directory not provided
I20260925 04:57:45.036444 19877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:57:45.036751 19877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:57:45.037709 19796 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.29.188" port: 43147 } attempt: 3
I20260925 04:57:45.038822 19799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:45.038372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:43147 (local address 127.0.0.1:51844)
0925 04:57:45.038571 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:45.038665 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:43147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 04:57:45.039880 19799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:45.039425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:43147 (local address 127.0.0.1:51848)
0925 04:57:45.039635 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:45.039698 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:43147: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260925 04:57:45.040203 19796 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.29.188:43147: Network error: Client connection negotiation failed: client connection to 127.6.29.188:43147: connect: Connection refused (error 111)
I20260925 04:57:45.040292 19877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-2/data/instance:
uuid: "cc6ba7be030a408795356dd0233f701b"
format_stamp: "Formatted at 2026-09-25 04:57:45 on dist-test-slave-xzjr"
server_key: "bfc50588042d421db157c6cd1d673c14"
server_key_iv: "d687d87b7ae58fdf29e4439f36653433"
server_key_version: "encryptionkey@0"
I20260925 04:57:45.041165 19877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-2/wal/instance:
uuid: "cc6ba7be030a408795356dd0233f701b"
format_stamp: "Formatted at 2026-09-25 04:57:45 on dist-test-slave-xzjr"
server_key: "bfc50588042d421db157c6cd1d673c14"
server_key_iv: "d687d87b7ae58fdf29e4439f36653433"
server_key_version: "encryptionkey@0"
I20260925 04:57:45.046546 19877 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 04:57:45.050233 19892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:45.051671 19877 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 04:57:45.051909 19877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-2/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-2/wal
uuid: "cc6ba7be030a408795356dd0233f701b"
format_stamp: "Formatted at 2026-09-25 04:57:45 on dist-test-slave-xzjr"
server_key: "bfc50588042d421db157c6cd1d673c14"
server_key_iv: "d687d87b7ae58fdf29e4439f36653433"
server_key_version: "encryptionkey@0"
I20260925 04:57:45.052119 19877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:57:45.070514 19877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:57:45.075126 19877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:57:45.075440 19877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:57:45.095280 19877 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.188:43147
I20260925 04:57:45.095314 19943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.188:43147 every 8 connection(s)
I20260925 04:57:45.096814 19877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-2/data/info.pb
I20260925 04:57:45.100874 19944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:57:45.106777 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 19877
I20260925 04:57:45.107170 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790312053666503-6262-0/minicluster-data/master-2/wal/instance
I20260925 04:57:45.107748 6262 external_mini_cluster.cc:1428] Setting key 95ef2fa22e0768379b7dece7374d163e
I20260925 04:57:45.108031 19944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40925 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 04:57:45 dist-test-slave-xzjr krb5kdc[19716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312265, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.188@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
Sep 25 04:57:45 dist-test-slave-xzjr krb5kdc[19716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.188@KRBTEST.COM
I20260925 04:57:45.130010 19800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:45.112013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42384 (local address 127.6.29.190:40925)
0925 04:57:45.112265 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:45.112274 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:45.112851 (+ 577us) server_negotiation.cc:407] Connection header received
0925 04:57:45.114643 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:45.114647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:45.114718 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:45.114844 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:45.116354 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:45.117099 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:45.118929 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:45.119620 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:45.119789 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:45.123823 (+ 4034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:45.123830 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:45.123843 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:45.123879 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:45.126559 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:45.127407 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:45.127413 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:45.127416 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:45.127564 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:45.128077 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:45.128088 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:45.128092 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:45.128434 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:45.128616 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:45.129381 (+ 765us) server_negotiation.cc:299] Negotiation successful
0925 04:57:45.129628 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 04:57:45.131182 19948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:45.111978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:40925 (local address 127.0.0.1:42384)
0925 04:57:45.112683 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:45.112730 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 04:57:45.114323 (+ 1593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:57:45.114909 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:57:45.114915 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:57:45.115673 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:57:45.116148 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:45.116171 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:45.117268 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:45.117273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:45.118771 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:45.118783 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:45.119908 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:45.119914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:45.120194 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:45.121073 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:57:45.121097 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:57:45.123592 (+ 2495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:57:45.126692 (+ 3100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:45.126705 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:45.126719 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:45.127280 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:45.127680 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:45.127685 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:45.127689 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:45.127939 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:45.128587 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:57:45.128596 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:57:45.128937 (+ 341us) client_negotiation.cc:769] Sending connection context
0925 04:57:45.129219 (+ 282us) client_negotiation.cc:240] Negotiation successful
0925 04:57:45.129760 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":277,"threads_started":1}
I20260925 04:57:45.131199 19947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:45.110044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51852 (local address 127.6.29.188:43147)
0925 04:57:45.111715 (+ 1671us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:45.111737 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:45.112503 (+ 766us) server_negotiation.cc:407] Connection header received
0925 04:57:45.114323 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:45.114354 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:45.114683 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:45.114909 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:45.116464 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:45.117995 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:45.119803 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:45.120644 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:45.120778 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:45.125195 (+ 4417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:45.125204 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:45.125217 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:45.125247 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:45.128197 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:45.129024 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:45.129031 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:45.129034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:45.129164 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:45.129534 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:45.129544 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:45.129546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:45.129901 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:45.130198 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:45.130552 (+ 354us) server_negotiation.cc:299] Negotiation successful
0925 04:57:45.130795 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":1376,"thread_start_us":1111,"threads_started":1}
I20260925 04:57:45.133263 19944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.29.189" port: 40393 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 04:57:45 dist-test-slave-xzjr krb5kdc[19716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312265, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.188@KRBTEST.COM for kudu/127.6.29.189@KRBTEST.COM
Sep 25 04:57:45 dist-test-slave-xzjr krb5kdc[19716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:57:45.147992 19948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:45.134089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:40393 (local address 127.0.0.1:36036)
0925 04:57:45.134236 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:45.134254 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:57:45.134374 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:57:45.134953 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:57:45.134957 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:57:45.135459 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:57:45.135662 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:45.135674 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:45.136791 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:45.136795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:45.137809 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:45.137824 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:45.138521 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:45.138526 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:45.138753 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:45.139392 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:57:45.139426 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:57:45.141770 (+ 2344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:57:45.145292 (+ 3522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:45.145297 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:45.145301 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:45.145848 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:45.146230 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:45.146235 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:45.146239 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:45.146384 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:45.146929 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:57:45.146934 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:57:45.147102 (+ 168us) client_negotiation.cc:769] Sending connection context
0925 04:57:45.147253 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 04:57:45.147575 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 04:57:45.148062 19875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:45.134098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36036 (local address 127.6.29.189:40393)
0925 04:57:45.134280 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:45.134285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:45.134306 (+ 21us) server_negotiation.cc:407] Connection header received
0925 04:57:45.134541 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:45.134550 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:45.134633 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:45.134758 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:45.135892 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:45.136631 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:45.137959 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:45.138376 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:45.138475 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:45.142050 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:45.142055 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:45.142058 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:45.142092 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:45.145154 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:45.145969 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:45.145977 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:45.145981 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:45.146112 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:45.146495 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:45.146501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:45.146505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:45.146768 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:45.146955 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:45.147359 (+ 404us) server_negotiation.cc:299] Negotiation successful
0925 04:57:45.147586 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 04:57:45.149355 19944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.29.188" port: 43147 } has no permanent_uuid. Determining permanent_uuid...
I20260925 04:57:45.150615 19800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:45.134817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42396 (local address 127.6.29.190:40925)
0925 04:57:45.134987 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:45.134990 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:45.135005 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:57:45.135045 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:45.135048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:45.135100 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:45.135171 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:45.136628 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:45.137401 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:45.138862 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:45.139455 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:45.139546 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:45.144185 (+ 4639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:45.144191 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:45.144194 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:45.144227 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:45.146838 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:45.147666 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:45.147670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:45.147672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:45.147861 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:45.149010 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:45.149015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:45.149018 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:45.149352 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:45.149674 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:45.150003 (+ 329us) server_negotiation.cc:299] Negotiation successful
0925 04:57:45.150264 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Sep 25 04:57:45 dist-test-slave-xzjr krb5kdc[19716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312265, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.188@KRBTEST.COM for kudu/127.6.29.188@KRBTEST.COM
Sep 25 04:57:45 dist-test-slave-xzjr krb5kdc[19716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.189@KRBTEST.COM
I20260925 04:57:45.164217 19947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:45.150200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51856 (local address 127.6.29.188:43147)
0925 04:57:45.150401 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:45.150407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:45.150620 (+ 213us) server_negotiation.cc:407] Connection header received
0925 04:57:45.150790 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:45.150795 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:45.150865 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:45.150973 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:45.151677 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:45.152715 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:45.153832 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:45.154263 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:45.154409 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:45.158024 (+ 3615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:45.158031 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:45.158039 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:45.158079 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:45.161424 (+ 3345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:45.162256 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:45.162263 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:45.162267 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:45.162411 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:45.162804 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:45.162811 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:45.162815 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:45.163099 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:45.163251 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:45.163583 (+ 332us) server_negotiation.cc:299] Negotiation successful
0925 04:57:45.163881 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":69}
I20260925 04:57:45.164606 19948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:45.150235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:43147 (local address 127.0.0.1:51856)
0925 04:57:45.150519 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:45.150538 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 04:57:45.150658 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:57:45.150991 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:57:45.150996 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:57:45.151414 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:57:45.151544 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:45.151553 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:45.152836 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:45.152841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:45.153698 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:45.153709 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:45.154403 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:45.154409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:45.154635 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:45.155288 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:57:45.155313 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:57:45.157822 (+ 2509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:57:45.161585 (+ 3763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:45.161591 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:45.161595 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:45.162121 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:45.162549 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:45.162556 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:45.162563 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:45.162697 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:45.163212 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:57:45.163217 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:57:45.163355 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 04:57:45.163509 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 04:57:45.164099 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":250}
I20260925 04:57:45.169052 19875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:45.152595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36050 (local address 127.6.29.189:40393)
0925 04:57:45.152765 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:45.152769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:45.152784 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:57:45.152826 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:45.152829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:45.152879 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:45.152988 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:45.154260 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:45.155099 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:45.157029 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:45.157532 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:45.157669 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:45.163229 (+ 5560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:45.163235 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:45.163239 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:45.163275 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:45.166252 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:45.166963 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:45.166969 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:45.166973 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:45.167116 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:45.167522 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:45.167527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:45.167532 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:45.167884 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:45.168190 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:45.168480 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 04:57:45.168715 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 04:57:45.169759 19944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc6ba7be030a408795356dd0233f701b: Bootstrap starting.
I20260925 04:57:45.173851 19944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc6ba7be030a408795356dd0233f701b: Neither blocks nor log segments found. Creating new log.
I20260925 04:57:45.174289 6262 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:57:45.175087 19944 log.cc:824] T 00000000000000000000000000000000 P cc6ba7be030a408795356dd0233f701b: Log is configured to *not* fsync() on all Append() calls
I20260925 04:57:45.177874 19944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc6ba7be030a408795356dd0233f701b: No bootstrap required, opened a new log
I20260925 04:57:45.190905 19944 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc6ba7be030a408795356dd0233f701b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c7addf0ebd4999826812bfa39e2c21" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40925 } } peers { permanent_uuid: "31cc47688e7943048a38fa3aed8cc54a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 40393 } } peers { permanent_uuid: "cc6ba7be030a408795356dd0233f701b" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 43147 } }
I20260925 04:57:45.191448 19944 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc6ba7be030a408795356dd0233f701b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:57:45.191599 19944 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc6ba7be030a408795356dd0233f701b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc6ba7be030a408795356dd0233f701b, State: Initialized, Role: FOLLOWER
I20260925 04:57:45.192433 19944 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc6ba7be030a408795356dd0233f701b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c7addf0ebd4999826812bfa39e2c21" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40925 } } peers { permanent_uuid: "31cc47688e7943048a38fa3aed8cc54a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 40393 } } peers { permanent_uuid: "cc6ba7be030a408795356dd0233f701b" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 43147 } }
I20260925 04:57:45.199599 19957 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc6ba7be030a408795356dd0233f701b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c7addf0ebd4999826812bfa39e2c21" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40925 } } peers { permanent_uuid: "31cc47688e7943048a38fa3aed8cc54a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 40393 } } peers { permanent_uuid: "cc6ba7be030a408795356dd0233f701b" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 43147 } } }
I20260925 04:57:45.202271 19957 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc6ba7be030a408795356dd0233f701b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:57:45.202131 19944 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc6ba7be030a408795356dd0233f701b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:57:45.203282 19875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:45.180589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36064 (local address 127.6.29.189:40393)
0925 04:57:45.180771 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:45.180777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:45.182984 (+ 2207us) server_negotiation.cc:407] Connection header received
0925 04:57:45.183597 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:45.183607 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:45.183717 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:45.183821 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:45.187612 (+ 3791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:45.188804 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:45.191160 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:45.191801 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:45.191920 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:45.195911 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:45.195923 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:45.195931 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:45.195985 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:45.199764 (+ 3779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:45.200705 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:45.200711 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:45.200715 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:45.200891 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:45.201368 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:45.201375 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:45.201380 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:45.201796 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:45.202045 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:45.202652 (+ 607us) server_negotiation.cc:299] Negotiation successful
0925 04:57:45.202904 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 04:57:45.206043 19947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:45.180634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51862 (local address 127.6.29.188:43147)
0925 04:57:45.180794 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:45.180801 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:45.187477 (+ 6676us) server_negotiation.cc:407] Connection header received
0925 04:57:45.187725 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:45.187735 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:45.187837 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:45.187916 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:45.190150 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:45.191220 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:45.193949 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:45.194759 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:45.194867 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:45.198756 (+ 3889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:45.198763 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:45.198767 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:45.198812 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:45.202515 (+ 3703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:45.203470 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:45.203475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:45.203478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:45.203666 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:45.204156 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:45.204162 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:45.204165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:45.204548 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:45.204888 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:45.205409 (+ 521us) server_negotiation.cc:299] Negotiation successful
0925 04:57:45.205623 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 04:57:45.209064 19800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:45.180045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42412 (local address 127.6.29.190:40925)
0925 04:57:45.180240 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:45.180248 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:45.187741 (+ 7493us) server_negotiation.cc:407] Connection header received
0925 04:57:45.188024 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:45.188030 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:45.188116 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:45.188186 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:45.190536 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:45.191659 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:45.194285 (+ 2626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:45.195194 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:45.195317 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:45.200857 (+ 5540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:45.200871 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:45.200876 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:45.200924 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:45.205012 (+ 4088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:45.206174 (+ 1162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:45.206182 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:45.206188 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:45.206423 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:45.206989 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:45.206996 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:45.207001 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:45.207416 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:45.207743 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:45.208468 (+ 725us) server_negotiation.cc:299] Negotiation successful
0925 04:57:45.208737 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 04:57:45.218341 19796 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.29.188" port: 43147 } attempt: 4
W20260925 04:57:45.221571 19968 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cc6ba7be030a408795356dd0233f701b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:57:45.221912 19968 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 25 04:57:45 dist-test-slave-xzjr krb5kdc[19716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312264, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for kudu/127.6.29.188@KRBTEST.COM
I20260925 04:57:45.217121 19877 master_runner.cc:186] Error getting master registration for 127.6.29.190:40925: OK, instance_id { permanent_uuid: "f4c7addf0ebd4999826812bfa39e2c21" instance_seqno: 1790312264074805 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 04:57:45.231630 19877 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 04:57:45.239138 19947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:45.219828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51878 (local address 127.6.29.188:43147)
0925 04:57:45.219994 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:45.220002 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:45.220021 (+ 19us) server_negotiation.cc:407] Connection header received
0925 04:57:45.220076 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:45.220081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:45.220175 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:45.220278 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:45.221438 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:45.222806 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:45.225466 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:45.226386 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:45.226537 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:45.231336 (+ 4799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:45.231343 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:45.231347 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:45.231401 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:45.235925 (+ 4524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:45.236792 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:45.236800 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:45.236805 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:45.237011 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:45.237452 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:45.237460 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:45.237464 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:45.237951 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:45.238095 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:45.238515 (+ 420us) server_negotiation.cc:299] Negotiation successful
0925 04:57:45.238770 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 04:57:45.240594 19799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:45.218985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:43147 (local address 127.0.0.1:51878)
0925 04:57:45.219469 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:45.219491 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 04:57:45.219603 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:57:45.220470 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:57:45.220476 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:57:45.221016 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:57:45.221252 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:45.221265 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:45.224210 (+ 2945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:45.224215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:45.225259 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:45.225273 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:45.226581 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:45.226587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:45.226779 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:45.227518 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:57:45.227561 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:57:45.231075 (+ 3514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:57:45.236077 (+ 5002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:45.236083 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:45.236088 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:45.236667 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:45.237127 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:45.237132 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:45.237136 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:45.237329 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:45.238069 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:57:45.238074 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:57:45.238261 (+ 187us) client_negotiation.cc:769] Sending connection context
0925 04:57:45.238419 (+ 158us) client_negotiation.cc:240] Negotiation successful
0925 04:57:45.240218 (+ 1799us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321}
I20260925 04:57:45.246719 19796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4c7addf0ebd4999826812bfa39e2c21: Bootstrap starting.
I20260925 04:57:45.251204 19796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4c7addf0ebd4999826812bfa39e2c21: Neither blocks nor log segments found. Creating new log.
I20260925 04:57:45.253135 19796 log.cc:824] T 00000000000000000000000000000000 P f4c7addf0ebd4999826812bfa39e2c21: Log is configured to *not* fsync() on all Append() calls
I20260925 04:57:45.256151 19796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4c7addf0ebd4999826812bfa39e2c21: No bootstrap required, opened a new log
I20260925 04:57:45.264302 19796 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4c7addf0ebd4999826812bfa39e2c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c7addf0ebd4999826812bfa39e2c21" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40925 } } peers { permanent_uuid: "31cc47688e7943048a38fa3aed8cc54a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 40393 } } peers { permanent_uuid: "cc6ba7be030a408795356dd0233f701b" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 43147 } }
I20260925 04:57:45.264652 19796 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4c7addf0ebd4999826812bfa39e2c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:57:45.264787 19796 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4c7addf0ebd4999826812bfa39e2c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4c7addf0ebd4999826812bfa39e2c21, State: Initialized, Role: FOLLOWER
I20260925 04:57:45.265421 19796 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4c7addf0ebd4999826812bfa39e2c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c7addf0ebd4999826812bfa39e2c21" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40925 } } peers { permanent_uuid: "31cc47688e7943048a38fa3aed8cc54a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 40393 } } peers { permanent_uuid: "cc6ba7be030a408795356dd0233f701b" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 43147 } }
I20260925 04:57:45.266376 19971 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c7addf0ebd4999826812bfa39e2c21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c7addf0ebd4999826812bfa39e2c21" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40925 } } peers { permanent_uuid: "31cc47688e7943048a38fa3aed8cc54a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 40393 } } peers { permanent_uuid: "cc6ba7be030a408795356dd0233f701b" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 43147 } } }
I20260925 04:57:45.266763 19971 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c7addf0ebd4999826812bfa39e2c21 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:57:45.267612 19796 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4c7addf0ebd4999826812bfa39e2c21 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 04:57:45.280563 19982 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f4c7addf0ebd4999826812bfa39e2c21: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 04:57:45.280815 19982 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 04:57:45.281325 19957 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc6ba7be030a408795356dd0233f701b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:57:45.281649 19957 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc6ba7be030a408795356dd0233f701b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c7addf0ebd4999826812bfa39e2c21" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40925 } } peers { permanent_uuid: "31cc47688e7943048a38fa3aed8cc54a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 40393 } } peers { permanent_uuid: "cc6ba7be030a408795356dd0233f701b" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 43147 } }
I20260925 04:57:45.283300 19957 leader_election.cc:288] T 00000000000000000000000000000000 P cc6ba7be030a408795356dd0233f701b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f4c7addf0ebd4999826812bfa39e2c21 (127.6.29.190:40925), 31cc47688e7943048a38fa3aed8cc54a (127.6.29.189:40393)
I20260925 04:57:45.287038 19771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cc6ba7be030a408795356dd0233f701b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c7addf0ebd4999826812bfa39e2c21" is_pre_election: true
I20260925 04:57:45.287709 19771 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f4c7addf0ebd4999826812bfa39e2c21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc6ba7be030a408795356dd0233f701b in term 0.
I20260925 04:57:45.288681 19896 leader_election.cc:302] T 00000000000000000000000000000000 P cc6ba7be030a408795356dd0233f701b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cc6ba7be030a408795356dd0233f701b, f4c7addf0ebd4999826812bfa39e2c21; no voters:
I20260925 04:57:45.290491 19957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc6ba7be030a408795356dd0233f701b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:57:45.290655 19957 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc6ba7be030a408795356dd0233f701b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:57:45.290771 19957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc6ba7be030a408795356dd0233f701b [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:57:45.278976 19729 master_runner.cc:186] Error getting master registration for 127.6.29.189:40393: OK, instance_id { permanent_uuid: "31cc47688e7943048a38fa3aed8cc54a" instance_seqno: 1790312264572068 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 04:57:45.292454 19729 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 04:57:45.294257 19957 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc6ba7be030a408795356dd0233f701b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c7addf0ebd4999826812bfa39e2c21" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40925 } } peers { permanent_uuid: "31cc47688e7943048a38fa3aed8cc54a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 40393 } } peers { permanent_uuid: "cc6ba7be030a408795356dd0233f701b" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 43147 } }
I20260925 04:57:45.295583 19771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cc6ba7be030a408795356dd0233f701b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4c7addf0ebd4999826812bfa39e2c21"
I20260925 04:57:45.295862 19771 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4c7addf0ebd4999826812bfa39e2c21 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:57:45.295933 19957 leader_election.cc:288] T 00000000000000000000000000000000 P cc6ba7be030a408795356dd0233f701b [CANDIDATE]: Term 1 election: Requested vote from peers f4c7addf0ebd4999826812bfa39e2c21 (127.6.29.190:40925), 31cc47688e7943048a38fa3aed8cc54a (127.6.29.189:40393)
I20260925 04:57:45.299609 19771 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f4c7addf0ebd4999826812bfa39e2c21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc6ba7be030a408795356dd0233f701b in term 1.
I20260925 04:57:45.300171 19896 leader_election.cc:302] T 00000000000000000000000000000000 P cc6ba7be030a408795356dd0233f701b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cc6ba7be030a408795356dd0233f701b, f4c7addf0ebd4999826812bfa39e2c21; no voters:
I20260925 04:57:45.300529 19957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc6ba7be030a408795356dd0233f701b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:57:45.301227 19957 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc6ba7be030a408795356dd0233f701b [term 1 LEADER]: Becoming Leader. State: Replica: cc6ba7be030a408795356dd0233f701b, State: Running, Role: LEADER
I20260925 04:57:45.301863 19957 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc6ba7be030a408795356dd0233f701b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c7addf0ebd4999826812bfa39e2c21" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40925 } } peers { permanent_uuid: "31cc47688e7943048a38fa3aed8cc54a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 40393 } } peers { permanent_uuid: "cc6ba7be030a408795356dd0233f701b" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 43147 } }
I20260925 04:57:45.303241 19870 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.29.188" port: 43147 } attempt: 5
W20260925 04:57:45.314105 19750 tablet.cc:2369] T 00000000000000000000000000000000 P f4c7addf0ebd4999826812bfa39e2c21: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 04:57:45 dist-test-slave-xzjr krb5kdc[19716](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312264, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.189@KRBTEST.COM for kudu/127.6.29.188@KRBTEST.COM
I20260925 04:57:45.316941 19984 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc6ba7be030a408795356dd0233f701b [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc6ba7be030a408795356dd0233f701b. Latest consensus state: current_term: 1 leader_uuid: "cc6ba7be030a408795356dd0233f701b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c7addf0ebd4999826812bfa39e2c21" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40925 } } peers { permanent_uuid: "31cc47688e7943048a38fa3aed8cc54a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 40393 } } peers { permanent_uuid: "cc6ba7be030a408795356dd0233f701b" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 43147 } } }
I20260925 04:57:45.317423 19984 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc6ba7be030a408795356dd0233f701b [sys.catalog]: This master's current role is: LEADER
I20260925 04:57:45.321175 19986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:57:45.324534 19986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:57:45.326238 19873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:45.303919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:43147 (local address 127.0.0.1:51886)
0925 04:57:45.304185 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:45.304207 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 04:57:45.304341 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:57:45.304927 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:57:45.304932 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:57:45.305475 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:57:45.305687 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:45.305702 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:45.308449 (+ 2747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:45.308458 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:45.310481 (+ 2023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:45.310505 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:45.312172 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:45.312178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:45.312396 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:45.313166 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:57:45.313220 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:57:45.317360 (+ 4140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:57:45.322611 (+ 5251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:45.322618 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:45.322622 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:45.323167 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:45.323848 (+ 681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:45.323854 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:45.323857 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:45.324040 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:45.325151 (+ 1111us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:57:45.325160 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:57:45.325346 (+ 186us) client_negotiation.cc:769] Sending connection context
0925 04:57:45.325495 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 04:57:45.325807 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
W20260925 04:57:45.327538 19898 tablet.cc:2369] T 00000000000000000000000000000000 P cc6ba7be030a408795356dd0233f701b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:57:45.328266 19947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:45.303917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51886 (local address 127.6.29.188:43147)
0925 04:57:45.304098 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:45.304105 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:45.304271 (+ 166us) server_negotiation.cc:407] Connection header received
0925 04:57:45.304471 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:45.304484 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:45.304571 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:45.304662 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:45.305931 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:45.307696 (+ 1765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:45.310750 (+ 3054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:45.311738 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:45.311900 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:45.317703 (+ 5803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:45.317712 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:45.317722 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:45.317767 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:45.322426 (+ 4659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:45.323320 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:45.323356 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:45.323360 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:45.323634 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:45.324245 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:45.324252 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:45.324261 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:45.324941 (+ 680us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:45.326722 (+ 1781us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:45.326955 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 04:57:45.327406 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 04:57:45.334620 19870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31cc47688e7943048a38fa3aed8cc54a: Bootstrap starting.
I20260925 04:57:45.338879 19870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31cc47688e7943048a38fa3aed8cc54a: Neither blocks nor log segments found. Creating new log.
I20260925 04:57:45.340240 19870 log.cc:824] T 00000000000000000000000000000000 P 31cc47688e7943048a38fa3aed8cc54a: Log is configured to *not* fsync() on all Append() calls
I20260925 04:57:45.345960 19870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31cc47688e7943048a38fa3aed8cc54a: No bootstrap required, opened a new log
I20260925 04:57:45.351902 19771 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f4c7addf0ebd4999826812bfa39e2c21 [term 1 FOLLOWER]: Refusing update from remote peer cc6ba7be030a408795356dd0233f701b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:57:45.353046 19984 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cc6ba7be030a408795356dd0233f701b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4c7addf0ebd4999826812bfa39e2c21" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:57:45.357256 19870 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31cc47688e7943048a38fa3aed8cc54a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c7addf0ebd4999826812bfa39e2c21" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40925 } } peers { permanent_uuid: "31cc47688e7943048a38fa3aed8cc54a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 40393 } } peers { permanent_uuid: "cc6ba7be030a408795356dd0233f701b" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 43147 } }
I20260925 04:57:45.357698 19870 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31cc47688e7943048a38fa3aed8cc54a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:57:45.357810 19870 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31cc47688e7943048a38fa3aed8cc54a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31cc47688e7943048a38fa3aed8cc54a, State: Initialized, Role: FOLLOWER
I20260925 04:57:45.358644 19870 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31cc47688e7943048a38fa3aed8cc54a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c7addf0ebd4999826812bfa39e2c21" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40925 } } peers { permanent_uuid: "31cc47688e7943048a38fa3aed8cc54a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 40393 } } peers { permanent_uuid: "cc6ba7be030a408795356dd0233f701b" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 43147 } }
I20260925 04:57:45.365804 19990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31cc47688e7943048a38fa3aed8cc54a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c7addf0ebd4999826812bfa39e2c21" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40925 } } peers { permanent_uuid: "31cc47688e7943048a38fa3aed8cc54a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 40393 } } peers { permanent_uuid: "cc6ba7be030a408795356dd0233f701b" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 43147 } } }
I20260925 04:57:45.366247 19990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31cc47688e7943048a38fa3aed8cc54a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:57:45.368259 19870 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31cc47688e7943048a38fa3aed8cc54a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:57:45.368304 19844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cc6ba7be030a408795356dd0233f701b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31cc47688e7943048a38fa3aed8cc54a"
I20260925 04:57:45.368842 19844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31cc47688e7943048a38fa3aed8cc54a [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:57:45.372105 19845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cc6ba7be030a408795356dd0233f701b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31cc47688e7943048a38fa3aed8cc54a" is_pre_election: true
I20260925 04:57:45.375559 19844 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 31cc47688e7943048a38fa3aed8cc54a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc6ba7be030a408795356dd0233f701b in term 1.
I20260925 04:57:45.376506 19843 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 31cc47688e7943048a38fa3aed8cc54a [term 1 FOLLOWER]: Refusing update from remote peer cc6ba7be030a408795356dd0233f701b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:57:45.378679 19984 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cc6ba7be030a408795356dd0233f701b [LEADER]: Connected to new peer: Peer: permanent_uuid: "31cc47688e7943048a38fa3aed8cc54a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 40393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:57:45.388370 19971 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c7addf0ebd4999826812bfa39e2c21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc6ba7be030a408795356dd0233f701b. Latest consensus state: current_term: 1 leader_uuid: "cc6ba7be030a408795356dd0233f701b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c7addf0ebd4999826812bfa39e2c21" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40925 } } peers { permanent_uuid: "31cc47688e7943048a38fa3aed8cc54a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 40393 } } peers { permanent_uuid: "cc6ba7be030a408795356dd0233f701b" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 43147 } } }
I20260925 04:57:45.388816 19971 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c7addf0ebd4999826812bfa39e2c21 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:57:45.392397 19988 mvcc.cc:205] Tried to move back new op lower bound from 7333119038860529664 to 7333119038687240192. Current Snapshot: MvccSnapshot[applied={T|T < 7333119038860529664}]
I20260925 04:57:45.394866 19989 mvcc.cc:205] Tried to move back new op lower bound from 7333119038860529664 to 7333119038687240192. Current Snapshot: MvccSnapshot[applied={T|T < 7333119038860529664}]
I20260925 04:57:45.403106 19957 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc6ba7be030a408795356dd0233f701b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cc6ba7be030a408795356dd0233f701b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c7addf0ebd4999826812bfa39e2c21" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40925 } } peers { permanent_uuid: "31cc47688e7943048a38fa3aed8cc54a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 40393 } } peers { permanent_uuid: "cc6ba7be030a408795356dd0233f701b" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 43147 } } }
I20260925 04:57:45.403589 19957 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc6ba7be030a408795356dd0233f701b [sys.catalog]: This master's current role is: LEADER
I20260925 04:57:45.408851 19971 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4c7addf0ebd4999826812bfa39e2c21 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cc6ba7be030a408795356dd0233f701b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c7addf0ebd4999826812bfa39e2c21" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40925 } } peers { permanent_uuid: "31cc47688e7943048a38fa3aed8cc54a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 40393 } } peers { permanent_uuid: "cc6ba7be030a408795356dd0233f701b" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 43147 } } }
I20260925 04:57:45.409190 19971 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4c7addf0ebd4999826812bfa39e2c21 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:57:45.409797 19986 catalog_manager.cc:1347] Generated new cluster ID: 78f789c444cc42f2bdf887be6d803858
I20260925 04:57:45.409904 19986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:57:45.416567 19990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31cc47688e7943048a38fa3aed8cc54a [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc6ba7be030a408795356dd0233f701b. Latest consensus state: current_term: 1 leader_uuid: "cc6ba7be030a408795356dd0233f701b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c7addf0ebd4999826812bfa39e2c21" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40925 } } peers { permanent_uuid: "31cc47688e7943048a38fa3aed8cc54a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 40393 } } peers { permanent_uuid: "cc6ba7be030a408795356dd0233f701b" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 43147 } } }
I20260925 04:57:45.416909 19990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31cc47688e7943048a38fa3aed8cc54a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:57:45.418399 19984 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc6ba7be030a408795356dd0233f701b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cc6ba7be030a408795356dd0233f701b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c7addf0ebd4999826812bfa39e2c21" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40925 } } peers { permanent_uuid: "31cc47688e7943048a38fa3aed8cc54a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 40393 } } peers { permanent_uuid: "cc6ba7be030a408795356dd0233f701b" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 43147 } } }
I20260925 04:57:45.418754 19984 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc6ba7be030a408795356dd0233f701b [sys.catalog]: This master's current role is: LEADER
I20260925 04:57:45.426254 19986 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:57:45.428438 19986 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:57:45.431529 19990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31cc47688e7943048a38fa3aed8cc54a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cc6ba7be030a408795356dd0233f701b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c7addf0ebd4999826812bfa39e2c21" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40925 } } peers { permanent_uuid: "31cc47688e7943048a38fa3aed8cc54a" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 40393 } } peers { permanent_uuid: "cc6ba7be030a408795356dd0233f701b" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 43147 } } }
I20260925 04:57:45.431814 19990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31cc47688e7943048a38fa3aed8cc54a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 04:57:45.451351 19986 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc6ba7be030a408795356dd0233f701b: Generated new TSK 0
I20260925 04:57:45.452950 19986 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:57:45.465803 20007 catalog_manager.cc:1259] Loaded cluster ID: 78f789c444cc42f2bdf887be6d803858
I20260925 04:57:45.465999 20007 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 31cc47688e7943048a38fa3aed8cc54a: loading cluster ID for follower catalog manager: success
I20260925 04:57:45.470165 20007 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 31cc47688e7943048a38fa3aed8cc54a: acquiring CA information for follower catalog manager: success
I20260925 04:57:45.473066 20007 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 31cc47688e7943048a38fa3aed8cc54a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 04:57:46.282868 19982 catalog_manager.cc:1259] Loaded cluster ID: 78f789c444cc42f2bdf887be6d803858
I20260925 04:57:46.282981 19982 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f4c7addf0ebd4999826812bfa39e2c21: loading cluster ID for follower catalog manager: success
I20260925 04:57:46.285391 19982 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f4c7addf0ebd4999826812bfa39e2c21: acquiring CA information for follower catalog manager: success
I20260925 04:57:46.287108 19982 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f4c7addf0ebd4999826812bfa39e2c21: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 04:57:46.610451 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 19729
I20260925 04:57:46.612365 19791 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:57:46.724263 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 19729
I20260925 04:57:46.745703 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 19803
I20260925 04:57:46.746817 19865 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:57:46.857358 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 19803
I20260925 04:57:46.877599 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 19877
I20260925 04:57:46.878753 19939 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:57:46.988705 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 19877
2026-09-25T04:57:47Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4701 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:57:47 dist-test-slave-xzjr krb5kdc[20018](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:57:47 dist-test-slave-xzjr krb5kdc[20018](info): set up 2 sockets
Sep 25 04:57:47 dist-test-slave-xzjr krb5kdc[20018](info): commencing operation
krb5kdc: starting...
W20260925 04:57:48.096752 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 04:57:48 dist-test-slave-xzjr krb5kdc[20018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:57:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:57:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.29.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
I20260925 04:57:48.336115 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:33143
--webserver_interface=127.6.29.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:38299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:33143
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:57:48.665035 20031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:57:48.665330 20031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:57:48.665498 20031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:57:48.672641 20031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:57:48.672752 20031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:57:48.672863 20031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:57:48.672971 20031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:57:48.673058 20031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:57:48.683079 20031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:38299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:33143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:33143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:57:48.684453 20031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:57:48.685779 20031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:57:48.697629 20036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:48.697701 20037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:48.697734 20039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:57:48.698597 20031 server_base.cc:1034] running on GCE node
I20260925 04:57:48.699391 20031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:57:48.700927 20031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:57:48.702163 20031 hybrid_clock.cc:648] HybridClock initialized: now 1790312268702117 us; error 59 us; skew 500 ppm
I20260925 04:57:48.702519 20031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:57:48 dist-test-slave-xzjr krb5kdc[20018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312268, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:57:48.705626 20031 init.cc:377] Logged in from keytab as kudu/127.6.29.190@KRBTEST.COM (short username kudu)
I20260925 04:57:48.707191 20031 webserver.cc:460] Webserver started at http://127.6.29.190:42349/ using document root <none> and password file <none>
I20260925 04:57:48.707955 20031 fs_manager.cc:359] Metadata directory not provided
I20260925 04:57:48.708101 20031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:57:48.708418 20031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:57:48.711473 20031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "3367fc4522b24ce0976d89d0d320d4b1"
format_stamp: "Formatted at 2026-09-25 04:57:48 on dist-test-slave-xzjr"
server_key: "451e15f156243d83934bd1e3c9275bc4"
server_key_iv: "f9834549ed681086e1097b1b62344874"
server_key_version: "encryptionkey@0"
I20260925 04:57:48.712359 20031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "3367fc4522b24ce0976d89d0d320d4b1"
format_stamp: "Formatted at 2026-09-25 04:57:48 on dist-test-slave-xzjr"
server_key: "451e15f156243d83934bd1e3c9275bc4"
server_key_iv: "f9834549ed681086e1097b1b62344874"
server_key_version: "encryptionkey@0"
I20260925 04:57:48.717549 20031 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 04:57:48.721109 20046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:48.722857 20031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:57:48.723085 20031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "3367fc4522b24ce0976d89d0d320d4b1"
format_stamp: "Formatted at 2026-09-25 04:57:48 on dist-test-slave-xzjr"
server_key: "451e15f156243d83934bd1e3c9275bc4"
server_key_iv: "f9834549ed681086e1097b1b62344874"
server_key_version: "encryptionkey@0"
I20260925 04:57:48.723356 20031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:57:48.738688 20031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:57:48.743168 20031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:57:48.743466 20031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:57:48.762377 20031 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:33143
I20260925 04:57:48.762494 20097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:33143 every 8 connection(s)
I20260925 04:57:48.763952 20031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:57:48.767918 20098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:57:48.768502 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 20031
I20260925 04:57:48.768718 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 04:57:48.769362 6262 external_mini_cluster.cc:1428] Setting key 6f343fdb7c0e17a9b961fbc9e30d71ee
I20260925 04:57:48.781984 20098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3367fc4522b24ce0976d89d0d320d4b1: Bootstrap starting.
I20260925 04:57:48.785085 20098 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3367fc4522b24ce0976d89d0d320d4b1: Neither blocks nor log segments found. Creating new log.
I20260925 04:57:48.786355 20098 log.cc:824] T 00000000000000000000000000000000 P 3367fc4522b24ce0976d89d0d320d4b1: Log is configured to *not* fsync() on all Append() calls
Sep 25 04:57:48 dist-test-slave-xzjr krb5kdc[20018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:57:48.789445 20098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3367fc4522b24ce0976d89d0d320d4b1: No bootstrap required, opened a new log
I20260925 04:57:48.799016 20101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:48.771189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44026 (local address 127.6.29.190:33143)
0925 04:57:48.772046 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:48.772058 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:48.772098 (+ 40us) server_negotiation.cc:407] Connection header received
0925 04:57:48.774018 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:48.774044 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:48.774448 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:48.774800 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:48.776657 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:48.778057 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:48.780078 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:48.780782 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:48.780920 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:48.789673 (+ 8753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:48.789685 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:48.789704 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:48.789759 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:48.793953 (+ 4194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:48.794771 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:48.794780 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:48.794788 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:48.795026 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:48.795424 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:48.795429 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:48.795434 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:48.796240 (+ 806us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:48.796558 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:48.796981 (+ 423us) server_negotiation.cc:299] Negotiation successful
0925 04:57:48.797482 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":379,"threads_started":1}
I20260925 04:57:48.800870 20098 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3367fc4522b24ce0976d89d0d320d4b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3367fc4522b24ce0976d89d0d320d4b1" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33143 } }
I20260925 04:57:48.801328 20098 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3367fc4522b24ce0976d89d0d320d4b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:57:48.801487 20098 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3367fc4522b24ce0976d89d0d320d4b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3367fc4522b24ce0976d89d0d320d4b1, State: Initialized, Role: FOLLOWER
I20260925 04:57:48.802106 20098 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3367fc4522b24ce0976d89d0d320d4b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3367fc4522b24ce0976d89d0d320d4b1" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33143 } }
I20260925 04:57:48.802373 20098 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3367fc4522b24ce0976d89d0d320d4b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:57:48.802510 20098 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3367fc4522b24ce0976d89d0d320d4b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:57:48.802695 20098 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3367fc4522b24ce0976d89d0d320d4b1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:57:48.804101 6262 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:57:48.805227 20098 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3367fc4522b24ce0976d89d0d320d4b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3367fc4522b24ce0976d89d0d320d4b1" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33143 } }
I20260925 04:57:48.805727 20098 leader_election.cc:302] T 00000000000000000000000000000000 P 3367fc4522b24ce0976d89d0d320d4b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3367fc4522b24ce0976d89d0d320d4b1; no voters:
I20260925 04:57:48.806330 20098 leader_election.cc:288] T 00000000000000000000000000000000 P 3367fc4522b24ce0976d89d0d320d4b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:57:48.806483 20103 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3367fc4522b24ce0976d89d0d320d4b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:57:48.808452 20098 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3367fc4522b24ce0976d89d0d320d4b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:57:48.812084 20103 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3367fc4522b24ce0976d89d0d320d4b1 [term 1 LEADER]: Becoming Leader. State: Replica: 3367fc4522b24ce0976d89d0d320d4b1, State: Running, Role: LEADER
I20260925 04:57:48.812618 20103 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3367fc4522b24ce0976d89d0d320d4b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3367fc4522b24ce0976d89d0d320d4b1" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33143 } }
I20260925 04:57:48.815421 20104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3367fc4522b24ce0976d89d0d320d4b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3367fc4522b24ce0976d89d0d320d4b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3367fc4522b24ce0976d89d0d320d4b1" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33143 } } }
I20260925 04:57:48.815820 20104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3367fc4522b24ce0976d89d0d320d4b1 [sys.catalog]: This master's current role is: LEADER
I20260925 04:57:48.815500 20105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3367fc4522b24ce0976d89d0d320d4b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3367fc4522b24ce0976d89d0d320d4b1. Latest consensus state: current_term: 1 leader_uuid: "3367fc4522b24ce0976d89d0d320d4b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3367fc4522b24ce0976d89d0d320d4b1" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33143 } } }
I20260925 04:57:48.816193 20105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3367fc4522b24ce0976d89d0d320d4b1 [sys.catalog]: This master's current role is: LEADER
I20260925 04:57:48.817970 20114 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:57:48.822470 20114 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:57:48.831131 20114 catalog_manager.cc:1347] Generated new cluster ID: ff06d0de048440a0aa128f05fb15c25a
I20260925 04:57:48.831269 20114 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:57:48.849478 20114 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:57:48.850914 20114 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:57:48.866400 20114 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3367fc4522b24ce0976d89d0d320d4b1: Generated new TSK 0
I20260925 04:57:48.867322 20114 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:57:48.916566 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 20031
I20260925 04:57:48.917754 20093 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:57:49.023754 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 20031
2026-09-25T04:57:49Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2037 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:57:49 dist-test-slave-xzjr krb5kdc[20126](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:57:49 dist-test-slave-xzjr krb5kdc[20126](info): set up 2 sockets
Sep 25 04:57:49 dist-test-slave-xzjr krb5kdc[20126](info): commencing operation
krb5kdc: starting...
W20260925 04:57:50.144775 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.005s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 04:57:50 dist-test-slave-xzjr krb5kdc[20126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:57:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:57:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.29.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
I20260925 04:57:50.396167 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:39755
--webserver_interface=127.6.29.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:37825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:39755
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:57:50.737366 20139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:57:50.737706 20139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:57:50.737867 20139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:57:50.745299 20139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:57:50.745424 20139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:57:50.745479 20139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:57:50.745530 20139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:57:50.745574 20139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:57:50.755716 20139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:37825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:39755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:39755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:57:50.757117 20139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:57:50.758513 20139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:57:50.770938 20145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:50.770905 20144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:50.770859 20147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:57:50.771907 20139 server_base.cc:1034] running on GCE node
I20260925 04:57:50.772805 20139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:57:50.774370 20139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:57:50.775635 20139 hybrid_clock.cc:648] HybridClock initialized: now 1790312270775591 us; error 58 us; skew 500 ppm
I20260925 04:57:50.776098 20139 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:57:50 dist-test-slave-xzjr krb5kdc[20126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312270, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:57:50.779402 20139 init.cc:377] Logged in from keytab as kudu/127.6.29.190@KRBTEST.COM (short username kudu)
I20260925 04:57:50.781080 20139 webserver.cc:460] Webserver started at http://127.6.29.190:36581/ using document root <none> and password file <none>
I20260925 04:57:50.781840 20139 fs_manager.cc:359] Metadata directory not provided
I20260925 04:57:50.782008 20139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:57:50.782325 20139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:57:50.785529 20139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "e28737f93785414ab258f58efc83f3c4"
format_stamp: "Formatted at 2026-09-25 04:57:50 on dist-test-slave-xzjr"
server_key: "9b6e52de35eadaa39c1d3a48011caaec"
server_key_iv: "bc6767f227e68d6cdee24806e1f192fb"
server_key_version: "encryptionkey@0"
I20260925 04:57:50.786350 20139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "e28737f93785414ab258f58efc83f3c4"
format_stamp: "Formatted at 2026-09-25 04:57:50 on dist-test-slave-xzjr"
server_key: "9b6e52de35eadaa39c1d3a48011caaec"
server_key_iv: "bc6767f227e68d6cdee24806e1f192fb"
server_key_version: "encryptionkey@0"
I20260925 04:57:50.792572 20139 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260925 04:57:50.796604 20154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:50.798074 20139 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:57:50.798326 20139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "e28737f93785414ab258f58efc83f3c4"
format_stamp: "Formatted at 2026-09-25 04:57:50 on dist-test-slave-xzjr"
server_key: "9b6e52de35eadaa39c1d3a48011caaec"
server_key_iv: "bc6767f227e68d6cdee24806e1f192fb"
server_key_version: "encryptionkey@0"
I20260925 04:57:50.798547 20139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:57:50.842779 20139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:57:50.847347 20139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:57:50.847666 20139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:57:50.867341 20139 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:39755
I20260925 04:57:50.867405 20205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:39755 every 8 connection(s)
I20260925 04:57:50.868924 20139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:57:50.870585 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 20139
I20260925 04:57:50.870828 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 04:57:50.871541 6262 external_mini_cluster.cc:1428] Setting key b14478f41fc0f089b63710622b3680c6
I20260925 04:57:50.873322 20206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 04:57:50 dist-test-slave-xzjr krb5kdc[20126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:57:50.888190 20206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4: Bootstrap starting.
I20260925 04:57:50.892740 20206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4: Neither blocks nor log segments found. Creating new log.
I20260925 04:57:50.893890 20206 log.cc:824] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4: Log is configured to *not* fsync() on all Append() calls
I20260925 04:57:50.896699 20209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:50.873679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49100 (local address 127.6.29.190:39755)
0925 04:57:50.874462 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:50.874473 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:50.874513 (+ 40us) server_negotiation.cc:407] Connection header received
0925 04:57:50.876175 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:50.876197 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:50.876583 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:50.876937 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:50.878870 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:50.880350 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:50.882246 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:50.882931 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:50.883060 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:50.888030 (+ 4970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:50.888041 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:50.888063 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:50.888113 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:50.892165 (+ 4052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:50.893047 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:50.893059 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:50.893066 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:50.893259 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:50.893695 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:50.893701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:50.893704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:50.894344 (+ 640us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:50.894696 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:50.895161 (+ 465us) server_negotiation.cc:299] Negotiation successful
0925 04:57:50.895601 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":343,"threads_started":1}
I20260925 04:57:50.897233 20206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4: No bootstrap required, opened a new log
I20260925 04:57:50.906126 20206 raft_consensus.cc:348] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28737f93785414ab258f58efc83f3c4" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39755 } }
I20260925 04:57:50.906539 20206 raft_consensus.cc:374] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:57:50.906649 20206 raft_consensus.cc:729] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e28737f93785414ab258f58efc83f3c4, State: Initialized, Role: FOLLOWER
I20260925 04:57:50.907209 20206 consensus_queue.cc:260] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28737f93785414ab258f58efc83f3c4" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39755 } }
I20260925 04:57:50.907474 20206 raft_consensus.cc:388] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:57:50.907619 20206 raft_consensus.cc:482] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:57:50.907847 20206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:57:50.910441 20206 raft_consensus.cc:504] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28737f93785414ab258f58efc83f3c4" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39755 } }
I20260925 04:57:50.910969 20206 leader_election.cc:302] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e28737f93785414ab258f58efc83f3c4; no voters:
I20260925 04:57:50.911626 20206 leader_election.cc:288] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:57:50.911772 20211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:57:50.912603 20211 raft_consensus.cc:686] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [term 1 LEADER]: Becoming Leader. State: Replica: e28737f93785414ab258f58efc83f3c4, State: Running, Role: LEADER
I20260925 04:57:50.913192 20211 consensus_queue.cc:237] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28737f93785414ab258f58efc83f3c4" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39755 } }
I20260925 04:57:50.914146 20206 sys_catalog.cc:564] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:57:50.916008 20212 sys_catalog.cc:455] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e28737f93785414ab258f58efc83f3c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28737f93785414ab258f58efc83f3c4" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39755 } } }
I20260925 04:57:50.916389 20212 sys_catalog.cc:458] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [sys.catalog]: This master's current role is: LEADER
I20260925 04:57:50.917276 20217 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:57:50.917080 20213 sys_catalog.cc:455] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e28737f93785414ab258f58efc83f3c4. Latest consensus state: current_term: 1 leader_uuid: "e28737f93785414ab258f58efc83f3c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28737f93785414ab258f58efc83f3c4" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39755 } } }
I20260925 04:57:50.921402 20213 sys_catalog.cc:458] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [sys.catalog]: This master's current role is: LEADER
I20260925 04:57:50.925173 20217 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:57:50.935877 20217 catalog_manager.cc:1347] Generated new cluster ID: 9763c0e1d58a45b6b6c6faaeb1946feb
I20260925 04:57:50.936002 20217 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:57:50.944540 20217 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:57:50.946516 20217 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:57:50.956440 20217 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4: Generated new TSK 0
I20260925 04:57:50.957322 20217 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.29.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab.
Entry for principal kudu/127.6.29.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab.
I20260925 04:57:51.024209 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.129:0
--local_ip_for_outbound_sockets=127.6.29.129
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:39755
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:37825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:57:51.357267 20232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:57:51.357546 20232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:57:51.357709 20232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:57:51.365129 20232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:57:51.365253 20232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:57:51.365500 20232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.129
I20260925 04:57:51.376094 20232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:37825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:39755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.129
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:57:51.377564 20232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:57:51.378991 20232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:57:51.392617 20238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:51.392578 20240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:51.392661 20237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:57:51.401656 20232 server_base.cc:1034] running on GCE node
I20260925 04:57:51.402462 20232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:57:51.403644 20232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:57:51.404889 20232 hybrid_clock.cc:648] HybridClock initialized: now 1790312271404835 us; error 46 us; skew 500 ppm
I20260925 04:57:51.405232 20232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:57:51 dist-test-slave-xzjr krb5kdc[20126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312271, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:57:51.408756 20232 init.cc:377] Logged in from keytab as kudu/127.6.29.129@KRBTEST.COM (short username kudu)
I20260925 04:57:51.410490 20232 webserver.cc:460] Webserver started at http://127.6.29.129:46169/ using document root <none> and password file <none>
I20260925 04:57:51.411298 20232 fs_manager.cc:359] Metadata directory not provided
I20260925 04:57:51.411458 20232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:57:51.411846 20232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:57:51.415134 20232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/data/instance:
uuid: "4a940a2e05694251b159ccbb2c4dda1c"
format_stamp: "Formatted at 2026-09-25 04:57:51 on dist-test-slave-xzjr"
server_key: "4afd8bcd8f07cdc64bd1749685dc7483"
server_key_iv: "69da25b82495cf436cc9170905d8fe96"
server_key_version: "encryptionkey@0"
I20260925 04:57:51.416077 20232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance:
uuid: "4a940a2e05694251b159ccbb2c4dda1c"
format_stamp: "Formatted at 2026-09-25 04:57:51 on dist-test-slave-xzjr"
server_key: "4afd8bcd8f07cdc64bd1749685dc7483"
server_key_iv: "69da25b82495cf436cc9170905d8fe96"
server_key_version: "encryptionkey@0"
I20260925 04:57:51.422291 20232 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260925 04:57:51.426256 20247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:51.428016 20232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:57:51.428224 20232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/wal
uuid: "4a940a2e05694251b159ccbb2c4dda1c"
format_stamp: "Formatted at 2026-09-25 04:57:51 on dist-test-slave-xzjr"
server_key: "4afd8bcd8f07cdc64bd1749685dc7483"
server_key_iv: "69da25b82495cf436cc9170905d8fe96"
server_key_version: "encryptionkey@0"
I20260925 04:57:51.428462 20232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:57:51.440508 20232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:57:51.445200 20232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:57:51.445546 20232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:57:51.446605 20232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:57:51.448130 20232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:57:51.448206 20232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:51.448362 20232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:57:51.448451 20232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:51.482739 20232 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.129:44167
I20260925 04:57:51.482761 20359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.129:44167 every 8 connection(s)
I20260925 04:57:51.484278 20232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
I20260925 04:57:51.488197 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 20232
I20260925 04:57:51.488462 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance
I20260925 04:57:51.489295 6262 external_mini_cluster.cc:1428] Setting key 60d7a1e7a52de7ec61fb5ebcaff65ea9
Sep 25 04:57:51 dist-test-slave-xzjr krb5kdc[20126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312271, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:57:51.508078 20362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:51.487191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:58145 (local address 127.6.29.190:39755)
0925 04:57:51.487732 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:51.487737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:51.489108 (+ 1371us) server_negotiation.cc:407] Connection header received
0925 04:57:51.491324 (+ 2216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:51.491329 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:51.491418 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:51.491546 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:51.493259 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:51.493970 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:51.495635 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:51.496090 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:51.496228 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:51.500961 (+ 4733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:51.500967 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:51.500971 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:51.501014 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:51.504402 (+ 3388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:51.505366 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:51.505371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:51.505373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:51.505514 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:51.506024 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:51.506034 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:51.506037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:51.506344 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:51.506512 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:51.507335 (+ 823us) server_negotiation.cc:299] Negotiation successful
0925 04:57:51.507664 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":262,"threads_started":1}
I20260925 04:57:51.509184 20363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:51.487858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.129:58145)
0925 04:57:51.488920 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:51.488962 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 04:57:51.490972 (+ 2010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:57:51.491885 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:57:51.491897 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:57:51.492470 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:57:51.493079 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:51.493098 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:51.494141 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:51.494149 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:51.495470 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:51.495483 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:51.496340 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:51.496348 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:51.496588 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:51.497331 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:57:51.497356 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:57:51.500619 (+ 3263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:57:51.504612 (+ 3993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:51.504624 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:51.504650 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:51.505187 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:51.505643 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:51.505649 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:51.505653 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:51.505884 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:51.506489 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:57:51.506499 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:57:51.506914 (+ 415us) client_negotiation.cc:769] Sending connection context
0925 04:57:51.507208 (+ 294us) client_negotiation.cc:240] Negotiation successful
0925 04:57:51.507895 (+ 687us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"thread_start_us":485,"threads_started":1}
I20260925 04:57:51.511088 20360 heartbeater.cc:347] Connected to a master server at 127.6.29.190:39755
I20260925 04:57:51.511463 20360 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.6.29.130@KRBTEST.COM; defaulting to no policy
I20260925 04:57:51.512351 20360 heartbeater.cc:511] Master 127.6.29.190:39755 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.130@KRBTEST.COM" created.
I20260925 04:57:51.515149 20171 ts_manager.cc:194] Registered new tserver with Master: 4a940a2e05694251b159ccbb2c4dda1c (127.6.29.129:44167)
I20260925 04:57:51.517722 20171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.129@KRBTEST.COM'} at 127.6.29.129:58145
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab.
Entry for principal kudu/127.6.29.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab.
I20260925 04:57:51.547133 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.130:0
--local_ip_for_outbound_sockets=127.6.29.130
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:39755
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:37825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:57:51.884850 20366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:57:51.885176 20366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:57:51.885350 20366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:57:51.892834 20366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:57:51.892989 20366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:57:51.893165 20366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.130
I20260925 04:57:51.903249 20366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:37825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:39755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.130
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:57:51.904639 20366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:57:51.905970 20366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:57:51.919396 20372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:51.919425 20374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:51.919451 20371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:57:51.925483 20366 server_base.cc:1034] running on GCE node
I20260925 04:57:51.926237 20366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:57:51.927292 20366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:57:51.928520 20366 hybrid_clock.cc:648] HybridClock initialized: now 1790312271928467 us; error 47 us; skew 500 ppm
I20260925 04:57:51.928856 20366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:57:51 dist-test-slave-xzjr krb5kdc[20126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312271, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:57:51.932037 20366 init.cc:377] Logged in from keytab as kudu/127.6.29.130@KRBTEST.COM (short username kudu)
I20260925 04:57:51.933828 20366 webserver.cc:460] Webserver started at http://127.6.29.130:39135/ using document root <none> and password file <none>
I20260925 04:57:51.934581 20366 fs_manager.cc:359] Metadata directory not provided
I20260925 04:57:51.934736 20366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:57:51.935045 20366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:57:51.938051 20366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/data/instance:
uuid: "e59fc77557484dd4a51d755847101947"
format_stamp: "Formatted at 2026-09-25 04:57:51 on dist-test-slave-xzjr"
server_key: "c38c87be4c7f1bb98772732cf809c71e"
server_key_iv: "c6ffd68bd8c9bf3d907b51b48104017a"
server_key_version: "encryptionkey@0"
I20260925 04:57:51.938834 20366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance:
uuid: "e59fc77557484dd4a51d755847101947"
format_stamp: "Formatted at 2026-09-25 04:57:51 on dist-test-slave-xzjr"
server_key: "c38c87be4c7f1bb98772732cf809c71e"
server_key_iv: "c6ffd68bd8c9bf3d907b51b48104017a"
server_key_version: "encryptionkey@0"
I20260925 04:57:51.944247 20366 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 04:57:51.947991 20381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:51.949504 20366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:57:51.949705 20366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/wal
uuid: "e59fc77557484dd4a51d755847101947"
format_stamp: "Formatted at 2026-09-25 04:57:51 on dist-test-slave-xzjr"
server_key: "c38c87be4c7f1bb98772732cf809c71e"
server_key_iv: "c6ffd68bd8c9bf3d907b51b48104017a"
server_key_version: "encryptionkey@0"
I20260925 04:57:51.949935 20366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:57:51.966892 20366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:57:51.971338 20366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:57:51.971594 20366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:57:51.972652 20366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:57:51.974181 20366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:57:51.974256 20366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:51.974347 20366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:57:51.974406 20366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:52.008006 20366 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.130:35163
I20260925 04:57:52.008046 20493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.130:35163 every 8 connection(s)
I20260925 04:57:52.009496 20366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
I20260925 04:57:52.010993 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 20366
I20260925 04:57:52.011173 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance
I20260925 04:57:52.011847 6262 external_mini_cluster.cc:1428] Setting key e9a6ad9466553193ad585906d223ed34
Sep 25 04:57:52 dist-test-slave-xzjr krb5kdc[20126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312271, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:57:52.033552 20496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:52.012408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:35579 (local address 127.6.29.190:39755)
0925 04:57:52.012939 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:52.012943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:52.014139 (+ 1196us) server_negotiation.cc:407] Connection header received
0925 04:57:52.016198 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:52.016204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:52.016305 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:52.016423 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:52.018019 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:52.018990 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:52.020767 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:52.021534 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:52.021671 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:52.025620 (+ 3949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:52.025638 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:52.025646 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:52.025701 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:52.029607 (+ 3906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:52.030442 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:52.030451 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:52.030457 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:52.030622 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:52.031063 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:52.031071 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:52.031078 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:52.031550 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:52.031714 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:52.032718 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0925 04:57:52.032995 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":254,"threads_started":1}
I20260925 04:57:52.034240 20497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:52.013091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.130:35579)
0925 04:57:52.013966 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:52.014022 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 04:57:52.015922 (+ 1900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:57:52.016680 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:57:52.016690 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:57:52.017248 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:57:52.017842 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:52.017860 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:52.019182 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:52.019192 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:52.020613 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:52.020625 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:52.021788 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:52.021793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:52.022012 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:52.022729 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:57:52.022755 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:57:52.025360 (+ 2605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:57:52.029777 (+ 4417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:52.029785 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:52.029798 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:52.030292 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:52.030732 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:52.030735 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:52.030737 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:52.030933 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:52.031683 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:57:52.031689 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:57:52.032218 (+ 529us) client_negotiation.cc:769] Sending connection context
0925 04:57:52.032568 (+ 350us) client_negotiation.cc:240] Negotiation successful
0925 04:57:52.032978 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":298,"threads_started":1}
I20260925 04:57:52.035835 20494 heartbeater.cc:347] Connected to a master server at 127.6.29.190:39755
I20260925 04:57:52.036203 20494 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.6.29.131@KRBTEST.COM; defaulting to no policy
I20260925 04:57:52.037120 20494 heartbeater.cc:511] Master 127.6.29.190:39755 requested a full tablet report, sending...
I20260925 04:57:52.038820 20171 ts_manager.cc:194] Registered new tserver with Master: e59fc77557484dd4a51d755847101947 (127.6.29.130:35163)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.131@KRBTEST.COM" created.
I20260925 04:57:52.040000 20171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.130@KRBTEST.COM'} at 127.6.29.130:35579
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab.
Entry for principal kudu/127.6.29.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab.
I20260925 04:57:52.067986 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.131:0
--local_ip_for_outbound_sockets=127.6.29.131
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:39755
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:37825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:57:52.408159 20500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:57:52.408463 20500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:57:52.408555 20500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:57:52.416056 20500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:57:52.416172 20500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:57:52.416285 20500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.131
I20260925 04:57:52.426235 20500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:37825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:39755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.131
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:57:52.427520 20500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:57:52.428855 20500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:57:52.442521 20506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:52.442487 20508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:52.442499 20505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:57:52.443333 20500 server_base.cc:1034] running on GCE node
I20260925 04:57:52.444092 20500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:57:52.445078 20500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:57:52.446322 20500 hybrid_clock.cc:648] HybridClock initialized: now 1790312272446242 us; error 68 us; skew 500 ppm
I20260925 04:57:52.446662 20500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:57:52 dist-test-slave-xzjr krb5kdc[20126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312272, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:57:52.450085 20500 init.cc:377] Logged in from keytab as kudu/127.6.29.131@KRBTEST.COM (short username kudu)
I20260925 04:57:52.451598 20500 webserver.cc:460] Webserver started at http://127.6.29.131:42127/ using document root <none> and password file <none>
I20260925 04:57:52.452360 20500 fs_manager.cc:359] Metadata directory not provided
I20260925 04:57:52.452499 20500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:57:52.452807 20500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:57:52.455997 20500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/data/instance:
uuid: "dff58da6112a4e8ba899f50028dd2b52"
format_stamp: "Formatted at 2026-09-25 04:57:52 on dist-test-slave-xzjr"
server_key: "b8ea3f81090e2492de138248abf6e460"
server_key_iv: "756f0ea0011f3845c45d63fb11b289ab"
server_key_version: "encryptionkey@0"
I20260925 04:57:52.456791 20500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance:
uuid: "dff58da6112a4e8ba899f50028dd2b52"
format_stamp: "Formatted at 2026-09-25 04:57:52 on dist-test-slave-xzjr"
server_key: "b8ea3f81090e2492de138248abf6e460"
server_key_iv: "756f0ea0011f3845c45d63fb11b289ab"
server_key_version: "encryptionkey@0"
I20260925 04:57:52.462191 20500 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 04:57:52.465960 20515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:52.467622 20500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:57:52.467851 20500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/wal
uuid: "dff58da6112a4e8ba899f50028dd2b52"
format_stamp: "Formatted at 2026-09-25 04:57:52 on dist-test-slave-xzjr"
server_key: "b8ea3f81090e2492de138248abf6e460"
server_key_iv: "756f0ea0011f3845c45d63fb11b289ab"
server_key_version: "encryptionkey@0"
I20260925 04:57:52.468088 20500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:57:52.514578 20500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:57:52.519315 20500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:57:52.519658 20500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:57:52.520776 20500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:57:52.521430 20360 heartbeater.cc:503] Master 127.6.29.190:39755 was elected leader, sending a full tablet report...
I20260925 04:57:52.522523 20500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:57:52.522598 20500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:52.522759 20500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:57:52.522812 20500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:52.556536 20500 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.131:46361
I20260925 04:57:52.556632 20627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.131:46361 every 8 connection(s)
I20260925 04:57:52.557870 20500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
I20260925 04:57:52.562391 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 20500
I20260925 04:57:52.562577 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance
I20260925 04:57:52.563241 6262 external_mini_cluster.cc:1428] Setting key 92c015ab23240eb8f439a86281dcce4a
Sep 25 04:57:52 dist-test-slave-xzjr krb5kdc[20126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312272, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:57:52.579638 20630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:52.560491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:34121 (local address 127.6.29.190:39755)
0925 04:57:52.560954 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:52.560959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:52.562032 (+ 1073us) server_negotiation.cc:407] Connection header received
0925 04:57:52.564399 (+ 2367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:52.564404 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:52.564497 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:52.564615 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:52.566176 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:52.566974 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:52.568621 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:52.569039 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:52.569176 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:52.573136 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:52.573141 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:52.573145 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:52.573182 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:52.576070 (+ 2888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:52.576934 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:52.576940 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:52.576943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:52.577110 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:52.577639 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:52.577643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:52.577647 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:52.577997 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:52.578173 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:52.579166 (+ 993us) server_negotiation.cc:299] Negotiation successful
0925 04:57:52.579357 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":222,"threads_started":1}
I20260925 04:57:52.580868 20631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:52.561048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.131:34121)
0925 04:57:52.561845 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:52.561890 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 04:57:52.564109 (+ 2219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:57:52.564863 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:57:52.564873 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:57:52.565412 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:57:52.566008 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:52.566026 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:52.567134 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:52.567138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:52.568476 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:52.568489 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:52.569267 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:52.569276 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:52.569529 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:52.570244 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:57:52.570268 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:57:52.572804 (+ 2536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:57:52.576282 (+ 3478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:52.576290 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:52.576305 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:52.576804 (+ 499us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:52.577260 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:52.577266 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:52.577270 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:52.577528 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:52.578172 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:57:52.578179 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:57:52.578725 (+ 546us) client_negotiation.cc:769] Sending connection context
0925 04:57:52.579049 (+ 324us) client_negotiation.cc:240] Negotiation successful
0925 04:57:52.579468 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":350,"threads_started":1}
I20260925 04:57:52.582579 20628 heartbeater.cc:347] Connected to a master server at 127.6.29.190:39755
I20260925 04:57:52.582880 20628 heartbeater.cc:464] Registering TS with master...
I20260925 04:57:52.583604 20628 heartbeater.cc:511] Master 127.6.29.190:39755 requested a full tablet report, sending...
I20260925 04:57:52.585161 20171 ts_manager.cc:194] Registered new tserver with Master: dff58da6112a4e8ba899f50028dd2b52 (127.6.29.131:46361)
I20260925 04:57:52.585964 20171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.131@KRBTEST.COM'} at 127.6.29.131:34121
I20260925 04:57:52.591120 6262 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:57:52.608439 20630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:52.595665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49104 (local address 127.6.29.190:39755)
0925 04:57:52.595896 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:52.595904 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:52.596323 (+ 419us) server_negotiation.cc:407] Connection header received
0925 04:57:52.596543 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:52.596547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:52.596612 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:52.596712 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:52.598079 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:52.598821 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:52.600195 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:52.600611 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:52.600738 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:52.603184 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:52.603190 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:52.603194 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:52.603233 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:52.605757 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:52.606429 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:52.606436 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:52.606443 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:52.606576 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:52.606949 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:52.606956 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:52.606961 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:52.607291 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:52.607537 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:52.607892 (+ 355us) server_negotiation.cc:299] Negotiation successful
0925 04:57:52.608123 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 04:57:52.612484 20171 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49104:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:57:52.620121 20171 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:57:52.656363 20644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:52.632481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40338 (local address 127.6.29.129:44167)
0925 04:57:52.647310 (+ 14829us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:52.647318 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:52.647345 (+ 27us) server_negotiation.cc:407] Connection header received
0925 04:57:52.647449 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:52.647455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:52.647686 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:52.647922 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:57:52.648610 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:52.650258 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:52.653605 (+ 3347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:52.655151 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:52.655305 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:52.655565 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:52.655835 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 04:57:52.655999 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14635,"thread_start_us":14490,"threads_started":1}
I20260925 04:57:52.656464 20641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:52.632399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:44167 (local address 127.0.0.1:40338)
0925 04:57:52.646914 (+ 14515us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:52.646934 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 04:57:52.647061 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:57:52.647922 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:57:52.647926 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:57:52.647955 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:57:52.648414 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:52.648425 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:52.650448 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:52.650454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:52.653434 (+ 2980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:52.653448 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:52.655350 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:52.655356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:52.655551 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:52.655606 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 04:57:52.655755 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 04:57:52.655999 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14376,"mutex_wait_us":122,"thread_start_us":13181,"threads_started":1}
I20260925 04:57:52.657852 20642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:52.632328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43828 (local address 127.6.29.130:35163)
0925 04:57:52.646917 (+ 14589us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:52.646930 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:52.646971 (+ 41us) server_negotiation.cc:407] Connection header received
0925 04:57:52.647085 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:52.647093 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:52.647354 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:52.647578 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:57:52.648529 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:52.650254 (+ 1725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:52.654387 (+ 4133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:52.655952 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:52.656066 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:52.656247 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:52.657352 (+ 1105us) server_negotiation.cc:299] Negotiation successful
0925 04:57:52.657528 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14353,"thread_start_us":14193,"threads_started":1}
I20260925 04:57:52.658419 20643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:52.632090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:35163 (local address 127.0.0.1:43828)
0925 04:57:52.644975 (+ 12885us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:52.645039 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 04:57:52.645310 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:57:52.647589 (+ 2279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:57:52.647596 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:57:52.647626 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:57:52.648133 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:52.648146 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:52.650448 (+ 2302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:52.650454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:52.654210 (+ 3756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:52.654223 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:52.656426 (+ 2203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:52.656435 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:52.656632 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:52.656666 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 04:57:52.656759 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 04:57:52.657138 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12484,"mutex_wait_us":1039,"thread_start_us":15054,"threads_started":1}
I20260925 04:57:52.661010 20646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:52.648383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.131:46361 (local address 127.0.0.1:44562)
0925 04:57:52.652930 (+ 4547us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:52.652950 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 04:57:52.653099 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:57:52.653870 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:57:52.653875 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:57:52.653906 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:57:52.654388 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:52.654401 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:52.656364 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:52.656372 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:52.658546 (+ 2174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:52.658562 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:52.660220 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:52.660227 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:52.660395 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:52.660429 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 04:57:52.660543 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 04:57:52.660673 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4393,"thread_start_us":4183,"threads_started":1}
I20260925 04:57:52.661110 20645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:52.647845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44562 (local address 127.6.29.131:46361)
0925 04:57:52.652526 (+ 4681us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:52.652536 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:52.653245 (+ 709us) server_negotiation.cc:407] Connection header received
0925 04:57:52.653353 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:52.653361 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:52.653626 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:52.653845 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:57:52.654576 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:52.656178 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:52.658739 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:52.660025 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:52.660170 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:52.660408 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:52.660597 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 04:57:52.660757 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4491,"thread_start_us":4334,"threads_started":1}
I20260925 04:57:52.660946 20295 tablet_service.cc:1467] Processing CreateTablet for tablet fdb56a2cbfff418f83d0ed1b33de7cb9 (DEFAULT_TABLE table=test-table [id=8da5def9dab64fe3a62fe41377c48b78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:57:52.662521 20295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdb56a2cbfff418f83d0ed1b33de7cb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:57:52.662209 20429 tablet_service.cc:1467] Processing CreateTablet for tablet fdb56a2cbfff418f83d0ed1b33de7cb9 (DEFAULT_TABLE table=test-table [id=8da5def9dab64fe3a62fe41377c48b78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:57:52.663668 20429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdb56a2cbfff418f83d0ed1b33de7cb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:57:52.664381 20563 tablet_service.cc:1467] Processing CreateTablet for tablet fdb56a2cbfff418f83d0ed1b33de7cb9 (DEFAULT_TABLE table=test-table [id=8da5def9dab64fe3a62fe41377c48b78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:57:52.665750 20563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdb56a2cbfff418f83d0ed1b33de7cb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:57:52.676777 20647 tablet_bootstrap.cc:492] T fdb56a2cbfff418f83d0ed1b33de7cb9 P dff58da6112a4e8ba899f50028dd2b52: Bootstrap starting.
I20260925 04:57:52.678874 20648 tablet_bootstrap.cc:492] T fdb56a2cbfff418f83d0ed1b33de7cb9 P 4a940a2e05694251b159ccbb2c4dda1c: Bootstrap starting.
I20260925 04:57:52.680718 20649 tablet_bootstrap.cc:492] T fdb56a2cbfff418f83d0ed1b33de7cb9 P e59fc77557484dd4a51d755847101947: Bootstrap starting.
I20260925 04:57:52.680749 20647 tablet_bootstrap.cc:654] T fdb56a2cbfff418f83d0ed1b33de7cb9 P dff58da6112a4e8ba899f50028dd2b52: Neither blocks nor log segments found. Creating new log.
I20260925 04:57:52.682533 20648 tablet_bootstrap.cc:654] T fdb56a2cbfff418f83d0ed1b33de7cb9 P 4a940a2e05694251b159ccbb2c4dda1c: Neither blocks nor log segments found. Creating new log.
I20260925 04:57:52.682552 20647 log.cc:824] T fdb56a2cbfff418f83d0ed1b33de7cb9 P dff58da6112a4e8ba899f50028dd2b52: Log is configured to *not* fsync() on all Append() calls
I20260925 04:57:52.683645 20649 tablet_bootstrap.cc:654] T fdb56a2cbfff418f83d0ed1b33de7cb9 P e59fc77557484dd4a51d755847101947: Neither blocks nor log segments found. Creating new log.
I20260925 04:57:52.683702 20648 log.cc:824] T fdb56a2cbfff418f83d0ed1b33de7cb9 P 4a940a2e05694251b159ccbb2c4dda1c: Log is configured to *not* fsync() on all Append() calls
I20260925 04:57:52.684851 20649 log.cc:824] T fdb56a2cbfff418f83d0ed1b33de7cb9 P e59fc77557484dd4a51d755847101947: Log is configured to *not* fsync() on all Append() calls
I20260925 04:57:52.686132 20647 tablet_bootstrap.cc:492] T fdb56a2cbfff418f83d0ed1b33de7cb9 P dff58da6112a4e8ba899f50028dd2b52: No bootstrap required, opened a new log
I20260925 04:57:52.686456 20647 ts_tablet_manager.cc:1397] T fdb56a2cbfff418f83d0ed1b33de7cb9 P dff58da6112a4e8ba899f50028dd2b52: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260925 04:57:52.687846 20649 tablet_bootstrap.cc:492] T fdb56a2cbfff418f83d0ed1b33de7cb9 P e59fc77557484dd4a51d755847101947: No bootstrap required, opened a new log
I20260925 04:57:52.688102 20649 ts_tablet_manager.cc:1397] T fdb56a2cbfff418f83d0ed1b33de7cb9 P e59fc77557484dd4a51d755847101947: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 04:57:52.688182 20648 tablet_bootstrap.cc:492] T fdb56a2cbfff418f83d0ed1b33de7cb9 P 4a940a2e05694251b159ccbb2c4dda1c: No bootstrap required, opened a new log
I20260925 04:57:52.688457 20648 ts_tablet_manager.cc:1397] T fdb56a2cbfff418f83d0ed1b33de7cb9 P 4a940a2e05694251b159ccbb2c4dda1c: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260925 04:57:52.695289 20649 raft_consensus.cc:348] T fdb56a2cbfff418f83d0ed1b33de7cb9 P e59fc77557484dd4a51d755847101947 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59fc77557484dd4a51d755847101947" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35163 } } peers { permanent_uuid: "4a940a2e05694251b159ccbb2c4dda1c" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 44167 } } peers { permanent_uuid: "dff58da6112a4e8ba899f50028dd2b52" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 46361 } }
I20260925 04:57:52.695706 20649 raft_consensus.cc:374] T fdb56a2cbfff418f83d0ed1b33de7cb9 P e59fc77557484dd4a51d755847101947 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:57:52.695842 20649 raft_consensus.cc:729] T fdb56a2cbfff418f83d0ed1b33de7cb9 P e59fc77557484dd4a51d755847101947 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59fc77557484dd4a51d755847101947, State: Initialized, Role: FOLLOWER
I20260925 04:57:52.696480 20649 consensus_queue.cc:260] T fdb56a2cbfff418f83d0ed1b33de7cb9 P e59fc77557484dd4a51d755847101947 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59fc77557484dd4a51d755847101947" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35163 } } peers { permanent_uuid: "4a940a2e05694251b159ccbb2c4dda1c" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 44167 } } peers { permanent_uuid: "dff58da6112a4e8ba899f50028dd2b52" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 46361 } }
I20260925 04:57:52.697527 20647 raft_consensus.cc:348] T fdb56a2cbfff418f83d0ed1b33de7cb9 P dff58da6112a4e8ba899f50028dd2b52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59fc77557484dd4a51d755847101947" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35163 } } peers { permanent_uuid: "4a940a2e05694251b159ccbb2c4dda1c" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 44167 } } peers { permanent_uuid: "dff58da6112a4e8ba899f50028dd2b52" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 46361 } }
I20260925 04:57:52.697976 20647 raft_consensus.cc:374] T fdb56a2cbfff418f83d0ed1b33de7cb9 P dff58da6112a4e8ba899f50028dd2b52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:57:52.698065 20647 raft_consensus.cc:729] T fdb56a2cbfff418f83d0ed1b33de7cb9 P dff58da6112a4e8ba899f50028dd2b52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff58da6112a4e8ba899f50028dd2b52, State: Initialized, Role: FOLLOWER
I20260925 04:57:52.698855 20649 ts_tablet_manager.cc:1428] T fdb56a2cbfff418f83d0ed1b33de7cb9 P e59fc77557484dd4a51d755847101947: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260925 04:57:52.698764 20647 consensus_queue.cc:260] T fdb56a2cbfff418f83d0ed1b33de7cb9 P dff58da6112a4e8ba899f50028dd2b52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59fc77557484dd4a51d755847101947" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35163 } } peers { permanent_uuid: "4a940a2e05694251b159ccbb2c4dda1c" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 44167 } } peers { permanent_uuid: "dff58da6112a4e8ba899f50028dd2b52" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 46361 } }
I20260925 04:57:52.699041 20494 heartbeater.cc:503] Master 127.6.29.190:39755 was elected leader, sending a full tablet report...
I20260925 04:57:52.699262 20648 raft_consensus.cc:348] T fdb56a2cbfff418f83d0ed1b33de7cb9 P 4a940a2e05694251b159ccbb2c4dda1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59fc77557484dd4a51d755847101947" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35163 } } peers { permanent_uuid: "4a940a2e05694251b159ccbb2c4dda1c" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 44167 } } peers { permanent_uuid: "dff58da6112a4e8ba899f50028dd2b52" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 46361 } }
I20260925 04:57:52.699751 20648 raft_consensus.cc:374] T fdb56a2cbfff418f83d0ed1b33de7cb9 P 4a940a2e05694251b159ccbb2c4dda1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:57:52.699882 20648 raft_consensus.cc:729] T fdb56a2cbfff418f83d0ed1b33de7cb9 P 4a940a2e05694251b159ccbb2c4dda1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a940a2e05694251b159ccbb2c4dda1c, State: Initialized, Role: FOLLOWER
I20260925 04:57:52.700012 20628 heartbeater.cc:503] Master 127.6.29.190:39755 was elected leader, sending a full tablet report...
I20260925 04:57:52.700686 20647 ts_tablet_manager.cc:1428] T fdb56a2cbfff418f83d0ed1b33de7cb9 P dff58da6112a4e8ba899f50028dd2b52: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260925 04:57:52.700564 20648 consensus_queue.cc:260] T fdb56a2cbfff418f83d0ed1b33de7cb9 P 4a940a2e05694251b159ccbb2c4dda1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59fc77557484dd4a51d755847101947" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35163 } } peers { permanent_uuid: "4a940a2e05694251b159ccbb2c4dda1c" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 44167 } } peers { permanent_uuid: "dff58da6112a4e8ba899f50028dd2b52" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 46361 } }
I20260925 04:57:52.703151 20648 ts_tablet_manager.cc:1428] T fdb56a2cbfff418f83d0ed1b33de7cb9 P 4a940a2e05694251b159ccbb2c4dda1c: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
W20260925 04:57:52.736757 20361 tablet.cc:2369] T fdb56a2cbfff418f83d0ed1b33de7cb9 P 4a940a2e05694251b159ccbb2c4dda1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:57:52.761745 20495 tablet.cc:2369] T fdb56a2cbfff418f83d0ed1b33de7cb9 P e59fc77557484dd4a51d755847101947: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:57:52.778975 20655 raft_consensus.cc:482] T fdb56a2cbfff418f83d0ed1b33de7cb9 P 4a940a2e05694251b159ccbb2c4dda1c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:57:52.779300 20655 raft_consensus.cc:504] T fdb56a2cbfff418f83d0ed1b33de7cb9 P 4a940a2e05694251b159ccbb2c4dda1c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59fc77557484dd4a51d755847101947" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35163 } } peers { permanent_uuid: "4a940a2e05694251b159ccbb2c4dda1c" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 44167 } } peers { permanent_uuid: "dff58da6112a4e8ba899f50028dd2b52" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 46361 } }
I20260925 04:57:52.781172 20655 leader_election.cc:288] T fdb56a2cbfff418f83d0ed1b33de7cb9 P 4a940a2e05694251b159ccbb2c4dda1c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e59fc77557484dd4a51d755847101947 (127.6.29.130:35163), dff58da6112a4e8ba899f50028dd2b52 (127.6.29.131:46361)
I20260925 04:57:52.788064 20642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:52.781783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:54113 (local address 127.6.29.130:35163)
0925 04:57:52.782040 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:52.782047 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:52.782250 (+ 203us) server_negotiation.cc:407] Connection header received
0925 04:57:52.782617 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:52.782624 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:52.782711 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:52.782778 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:57:52.783595 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:52.784387 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:52.786279 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:52.787040 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:52.787212 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:52.787291 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:52.787637 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 04:57:52.787789 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 04:57:52.788064 20645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:52.782486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:45329 (local address 127.6.29.131:46361)
0925 04:57:52.782678 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:52.782683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:52.782946 (+ 263us) server_negotiation.cc:407] Connection header received
0925 04:57:52.783059 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:52.783065 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:52.783143 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:52.783231 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:57:52.783832 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:52.784533 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:52.786302 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:52.787063 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:52.787242 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:52.787323 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:52.787648 (+ 325us) server_negotiation.cc:299] Negotiation successful
0925 04:57:52.787803 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 04:57:52.788210 20657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:52.782200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.131:46361 (local address 127.6.29.129:45329)
0925 04:57:52.782832 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:52.782846 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:57:52.782956 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:57:52.783268 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:57:52.783271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:57:52.783293 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:57:52.783646 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:52.783661 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:52.784677 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:52.784682 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:52.786143 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:52.786155 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:52.787291 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:52.787297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:52.787472 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:52.787505 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 04:57:52.787599 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 04:57:52.787808 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":65,"thread_start_us":308,"threads_started":1}
I20260925 04:57:52.788676 20656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:52.781469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:35163 (local address 127.6.29.129:54113)
0925 04:57:52.782128 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:52.782145 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:57:52.782389 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:57:52.782975 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:57:52.782982 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:57:52.783013 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:57:52.783446 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:52.783457 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:52.784514 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:52.784522 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:52.786136 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:52.786148 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:52.787253 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:52.787259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:52.787436 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:52.787500 (+ 64us) client_negotiation.cc:769] Sending connection context
0925 04:57:52.787599 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 04:57:52.788260 (+ 661us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"mutex_wait_us":100,"thread_start_us":302,"threads_started":1}
I20260925 04:57:52.789299 20583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdb56a2cbfff418f83d0ed1b33de7cb9" candidate_uuid: "4a940a2e05694251b159ccbb2c4dda1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff58da6112a4e8ba899f50028dd2b52" is_pre_election: true
I20260925 04:57:52.789870 20583 raft_consensus.cc:2454] T fdb56a2cbfff418f83d0ed1b33de7cb9 P dff58da6112a4e8ba899f50028dd2b52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a940a2e05694251b159ccbb2c4dda1c in term 0.
I20260925 04:57:52.789932 20449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdb56a2cbfff418f83d0ed1b33de7cb9" candidate_uuid: "4a940a2e05694251b159ccbb2c4dda1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59fc77557484dd4a51d755847101947" is_pre_election: true
I20260925 04:57:52.790452 20449 raft_consensus.cc:2454] T fdb56a2cbfff418f83d0ed1b33de7cb9 P e59fc77557484dd4a51d755847101947 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a940a2e05694251b159ccbb2c4dda1c in term 0.
I20260925 04:57:52.790516 20250 leader_election.cc:302] T fdb56a2cbfff418f83d0ed1b33de7cb9 P 4a940a2e05694251b159ccbb2c4dda1c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a940a2e05694251b159ccbb2c4dda1c, dff58da6112a4e8ba899f50028dd2b52; no voters:
I20260925 04:57:52.790956 20655 raft_consensus.cc:2781] T fdb56a2cbfff418f83d0ed1b33de7cb9 P 4a940a2e05694251b159ccbb2c4dda1c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:57:52.791111 20655 raft_consensus.cc:482] T fdb56a2cbfff418f83d0ed1b33de7cb9 P 4a940a2e05694251b159ccbb2c4dda1c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:57:52.791221 20655 raft_consensus.cc:3037] T fdb56a2cbfff418f83d0ed1b33de7cb9 P 4a940a2e05694251b159ccbb2c4dda1c [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:57:52.793430 20655 raft_consensus.cc:504] T fdb56a2cbfff418f83d0ed1b33de7cb9 P 4a940a2e05694251b159ccbb2c4dda1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59fc77557484dd4a51d755847101947" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35163 } } peers { permanent_uuid: "4a940a2e05694251b159ccbb2c4dda1c" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 44167 } } peers { permanent_uuid: "dff58da6112a4e8ba899f50028dd2b52" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 46361 } }
I20260925 04:57:52.794059 20655 leader_election.cc:288] T fdb56a2cbfff418f83d0ed1b33de7cb9 P 4a940a2e05694251b159ccbb2c4dda1c [CANDIDATE]: Term 1 election: Requested vote from peers e59fc77557484dd4a51d755847101947 (127.6.29.130:35163), dff58da6112a4e8ba899f50028dd2b52 (127.6.29.131:46361)
I20260925 04:57:52.794488 20449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdb56a2cbfff418f83d0ed1b33de7cb9" candidate_uuid: "4a940a2e05694251b159ccbb2c4dda1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59fc77557484dd4a51d755847101947"
I20260925 04:57:52.794560 20583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdb56a2cbfff418f83d0ed1b33de7cb9" candidate_uuid: "4a940a2e05694251b159ccbb2c4dda1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff58da6112a4e8ba899f50028dd2b52"
I20260925 04:57:52.794706 20449 raft_consensus.cc:3037] T fdb56a2cbfff418f83d0ed1b33de7cb9 P e59fc77557484dd4a51d755847101947 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:57:52.794790 20583 raft_consensus.cc:3037] T fdb56a2cbfff418f83d0ed1b33de7cb9 P dff58da6112a4e8ba899f50028dd2b52 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:57:52.798007 20583 raft_consensus.cc:2454] T fdb56a2cbfff418f83d0ed1b33de7cb9 P dff58da6112a4e8ba899f50028dd2b52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a940a2e05694251b159ccbb2c4dda1c in term 1.
I20260925 04:57:52.798007 20449 raft_consensus.cc:2454] T fdb56a2cbfff418f83d0ed1b33de7cb9 P e59fc77557484dd4a51d755847101947 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a940a2e05694251b159ccbb2c4dda1c in term 1.
I20260925 04:57:52.798406 20250 leader_election.cc:302] T fdb56a2cbfff418f83d0ed1b33de7cb9 P 4a940a2e05694251b159ccbb2c4dda1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a940a2e05694251b159ccbb2c4dda1c, dff58da6112a4e8ba899f50028dd2b52; no voters:
I20260925 04:57:52.798666 20655 raft_consensus.cc:2781] T fdb56a2cbfff418f83d0ed1b33de7cb9 P 4a940a2e05694251b159ccbb2c4dda1c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:57:52.799106 20655 raft_consensus.cc:686] T fdb56a2cbfff418f83d0ed1b33de7cb9 P 4a940a2e05694251b159ccbb2c4dda1c [term 1 LEADER]: Becoming Leader. State: Replica: 4a940a2e05694251b159ccbb2c4dda1c, State: Running, Role: LEADER
I20260925 04:57:52.799576 20655 consensus_queue.cc:237] T fdb56a2cbfff418f83d0ed1b33de7cb9 P 4a940a2e05694251b159ccbb2c4dda1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59fc77557484dd4a51d755847101947" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35163 } } peers { permanent_uuid: "4a940a2e05694251b159ccbb2c4dda1c" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 44167 } } peers { permanent_uuid: "dff58da6112a4e8ba899f50028dd2b52" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 46361 } }
I20260925 04:57:52.804307 20170 catalog_manager.cc:5495] T fdb56a2cbfff418f83d0ed1b33de7cb9 P 4a940a2e05694251b159ccbb2c4dda1c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a940a2e05694251b159ccbb2c4dda1c (127.6.29.129). New cstate: current_term: 1 leader_uuid: "4a940a2e05694251b159ccbb2c4dda1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e59fc77557484dd4a51d755847101947" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a940a2e05694251b159ccbb2c4dda1c" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 44167 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dff58da6112a4e8ba899f50028dd2b52" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 46361 } health_report { overall_health: UNKNOWN } } }
W20260925 04:57:52.809521 20629 tablet.cc:2369] T fdb56a2cbfff418f83d0ed1b33de7cb9 P dff58da6112a4e8ba899f50028dd2b52: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:57:52.839712 20644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:52.833714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40352 (local address 127.6.29.129:44167)
0925 04:57:52.833937 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:52.833945 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:52.834017 (+ 72us) server_negotiation.cc:407] Connection header received
0925 04:57:52.834263 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:52.834267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:52.834384 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:52.834550 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:57:52.834956 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:52.835684 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:52.837226 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:52.837867 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:52.838003 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:52.838587 (+ 584us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:57:52.839126 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:57:52.839303 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:52.839357 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 04:57:52.839444 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 04:57:52.848435 20583 raft_consensus.cc:1261] T fdb56a2cbfff418f83d0ed1b33de7cb9 P dff58da6112a4e8ba899f50028dd2b52 [term 1 FOLLOWER]: Refusing update from remote peer 4a940a2e05694251b159ccbb2c4dda1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:57:52.848433 20449 raft_consensus.cc:1261] T fdb56a2cbfff418f83d0ed1b33de7cb9 P e59fc77557484dd4a51d755847101947 [term 1 FOLLOWER]: Refusing update from remote peer 4a940a2e05694251b159ccbb2c4dda1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:57:52.849607 20658 consensus_queue.cc:1035] T fdb56a2cbfff418f83d0ed1b33de7cb9 P 4a940a2e05694251b159ccbb2c4dda1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff58da6112a4e8ba899f50028dd2b52" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 46361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:57:52.850026 20655 consensus_queue.cc:1035] T fdb56a2cbfff418f83d0ed1b33de7cb9 P 4a940a2e05694251b159ccbb2c4dda1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59fc77557484dd4a51d755847101947" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:57:52.870956 20662 mvcc.cc:205] Tried to move back new op lower bound from 7333119069575786496 to 7333119069391200256. Current Snapshot: MvccSnapshot[applied={T|T < 7333119069575786496}]
I20260925 04:57:52.877790 20169 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:57:52.878139 20169 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:57:52.882898 20169 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4: Sending DeleteTablet for 3 replicas of tablet fdb56a2cbfff418f83d0ed1b33de7cb9
I20260925 04:57:52.884392 20429 tablet_service.cc:1514] Processing DeleteTablet for tablet fdb56a2cbfff418f83d0ed1b33de7cb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:57:52 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:43828
I20260925 04:57:52.885383 20295 tablet_service.cc:1514] Processing DeleteTablet for tablet fdb56a2cbfff418f83d0ed1b33de7cb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:57:52 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:40338
I20260925 04:57:52.886935 20563 tablet_service.cc:1514] Processing DeleteTablet for tablet fdb56a2cbfff418f83d0ed1b33de7cb9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:57:52 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:44562
I20260925 04:57:52.887923 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 20232
I20260925 04:57:52.903115 20672 tablet_replica.cc:330] T fdb56a2cbfff418f83d0ed1b33de7cb9 P 4a940a2e05694251b159ccbb2c4dda1c: stopping tablet replica
I20260925 04:57:52.903709 20672 raft_consensus.cc:2229] T fdb56a2cbfff418f83d0ed1b33de7cb9 P 4a940a2e05694251b159ccbb2c4dda1c [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:57:52.904673 20672 raft_consensus.cc:2258] T fdb56a2cbfff418f83d0ed1b33de7cb9 P 4a940a2e05694251b159ccbb2c4dda1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:57:52.905620 20673 tablet_replica.cc:330] T fdb56a2cbfff418f83d0ed1b33de7cb9 P dff58da6112a4e8ba899f50028dd2b52: stopping tablet replica
I20260925 04:57:52.906245 20671 tablet_replica.cc:330] T fdb56a2cbfff418f83d0ed1b33de7cb9 P e59fc77557484dd4a51d755847101947: stopping tablet replica
Sep 25 04:57:52 dist-test-slave-xzjr krb5kdc[20126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.129@KRBTEST.COM
I20260925 04:57:52.906769 20673 raft_consensus.cc:2229] T fdb56a2cbfff418f83d0ed1b33de7cb9 P dff58da6112a4e8ba899f50028dd2b52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:57:52.907291 20673 raft_consensus.cc:2258] T fdb56a2cbfff418f83d0ed1b33de7cb9 P dff58da6112a4e8ba899f50028dd2b52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:57:52.908003 20672 ts_tablet_manager.cc:1905] T fdb56a2cbfff418f83d0ed1b33de7cb9 P 4a940a2e05694251b159ccbb2c4dda1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:57:52.907982 20671 raft_consensus.cc:2229] T fdb56a2cbfff418f83d0ed1b33de7cb9 P e59fc77557484dd4a51d755847101947 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:57:52.908454 20671 raft_consensus.cc:2258] T fdb56a2cbfff418f83d0ed1b33de7cb9 P e59fc77557484dd4a51d755847101947 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:57:52.910367 20671 ts_tablet_manager.cc:1905] T fdb56a2cbfff418f83d0ed1b33de7cb9 P e59fc77557484dd4a51d755847101947: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:57:52.912858 20673 ts_tablet_manager.cc:1905] T fdb56a2cbfff418f83d0ed1b33de7cb9 P dff58da6112a4e8ba899f50028dd2b52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:57:52.917538 20671 ts_tablet_manager.cc:1918] T fdb56a2cbfff418f83d0ed1b33de7cb9 P e59fc77557484dd4a51d755847101947: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:57:52.917729 20671 log.cc:1196] T fdb56a2cbfff418f83d0ed1b33de7cb9 P e59fc77557484dd4a51d755847101947: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/wal/wals/fdb56a2cbfff418f83d0ed1b33de7cb9
I20260925 04:57:52.918303 20671 ts_tablet_manager.cc:1939] T fdb56a2cbfff418f83d0ed1b33de7cb9 P e59fc77557484dd4a51d755847101947: Deleting consensus metadata
I20260925 04:57:52.918725 20644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:52.889100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40354 (local address 127.6.29.129:44167)
0925 04:57:52.889426 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:52.889433 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:52.892301 (+ 2868us) server_negotiation.cc:407] Connection header received
0925 04:57:52.892593 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:52.892599 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:52.892692 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:52.892790 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:52.897704 (+ 4914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:52.898987 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:52.901056 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:52.901783 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:52.901900 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:52.908368 (+ 6468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:52.908388 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:52.908407 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:52.908458 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:52.914180 (+ 5722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:52.915211 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:52.915220 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:52.915224 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:52.915400 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:52.915892 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:52.915899 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:52.915902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:52.916337 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:52.916666 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:52.918020 (+ 1354us) server_negotiation.cc:299] Negotiation successful
0925 04:57:52.918283 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204}
I20260925 04:57:52.919620 20355 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:57:52.920537 20157 catalog_manager.cc:4841] TS e59fc77557484dd4a51d755847101947 (127.6.29.130:35163): tablet fdb56a2cbfff418f83d0ed1b33de7cb9 (table test-table [id=8da5def9dab64fe3a62fe41377c48b78]) successfully deleted
I20260925 04:57:52.920851 20673 ts_tablet_manager.cc:1918] T fdb56a2cbfff418f83d0ed1b33de7cb9 P dff58da6112a4e8ba899f50028dd2b52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:57:52.921042 20673 log.cc:1196] T fdb56a2cbfff418f83d0ed1b33de7cb9 P dff58da6112a4e8ba899f50028dd2b52: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/wal/wals/fdb56a2cbfff418f83d0ed1b33de7cb9
I20260925 04:57:52.921720 20673 ts_tablet_manager.cc:1939] T fdb56a2cbfff418f83d0ed1b33de7cb9 P dff58da6112a4e8ba899f50028dd2b52: Deleting consensus metadata
I20260925 04:57:52.923825 20157 catalog_manager.cc:4841] TS dff58da6112a4e8ba899f50028dd2b52 (127.6.29.131:46361): tablet fdb56a2cbfff418f83d0ed1b33de7cb9 (table test-table [id=8da5def9dab64fe3a62fe41377c48b78]) successfully deleted
I20260925 04:57:53.023258 20672 ts_tablet_manager.cc:1918] T fdb56a2cbfff418f83d0ed1b33de7cb9 P 4a940a2e05694251b159ccbb2c4dda1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:57:53.023547 20672 log.cc:1196] T fdb56a2cbfff418f83d0ed1b33de7cb9 P 4a940a2e05694251b159ccbb2c4dda1c: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/wal/wals/fdb56a2cbfff418f83d0ed1b33de7cb9
I20260925 04:57:53.023813 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 20232
W20260925 04:57:53.041879 20158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.29.129:44167 (error 108)
I20260925 04:57:53.042188 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 20366
I20260925 04:57:53.043972 20646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:53.043474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:44167 (local address 127.0.0.1:40362)
0925 04:57:53.043629 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:53.043769 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.129:44167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 04:57:53.044204 20158 catalog_manager.cc:4584] TS 4a940a2e05694251b159ccbb2c4dda1c (127.6.29.129:44167): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fdb56a2cbfff418f83d0ed1b33de7cb9: Network error: Client connection negotiation failed: client connection to 127.6.29.129:44167: connect: Connection refused (error 111)
Sep 25 04:57:53 dist-test-slave-xzjr krb5kdc[20126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.130@KRBTEST.COM
I20260925 04:57:53.061992 20642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:53.043043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43838 (local address 127.6.29.130:35163)
0925 04:57:53.043212 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:53.043216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:53.043305 (+ 89us) server_negotiation.cc:407] Connection header received
0925 04:57:53.043466 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:53.043472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:53.043573 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:53.043681 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:53.045130 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:53.045842 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:53.047204 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:53.047589 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:53.047685 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:53.056392 (+ 8707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:53.056409 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:53.056430 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:53.056478 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:53.059189 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:53.059909 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:53.059924 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:53.059927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:53.060065 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:53.060485 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:53.060488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:53.060490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:53.060904 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:53.061149 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:53.061461 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 04:57:53.061624 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 04:57:53.062610 20489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:57:53.097831 20646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:53.097268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:44167 (local address 127.0.0.1:40368)
0925 04:57:53.097519 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:53.097639 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.129:44167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260925 04:57:53.098950 20646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:53.098544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:44167 (local address 127.0.0.1:40372)
0925 04:57:53.098705 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:53.098789 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.129:44167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 04:57:53.153858 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 20366
I20260925 04:57:53.159524 20646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:53.159072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:44167 (local address 127.0.0.1:40378)
0925 04:57:53.159283 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:53.159379 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.129:44167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 04:57:53.160605 20646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:53.160274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:44167 (local address 127.0.0.1:40394)
0925 04:57:53.160427 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:53.160500 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.129:44167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 04:57:53.172380 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 20500
Sep 25 04:57:53 dist-test-slave-xzjr krb5kdc[20126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.131@KRBTEST.COM
I20260925 04:57:53.187253 20645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:53.173217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44572 (local address 127.6.29.131:46361)
0925 04:57:53.173376 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:53.173381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:53.173399 (+ 18us) server_negotiation.cc:407] Connection header received
0925 04:57:53.173523 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:53.173527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:53.173599 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:53.173681 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:53.174909 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:53.175635 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:53.176937 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:53.177321 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:53.177413 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:53.181845 (+ 4432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:53.181865 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:53.181881 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:53.181925 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:53.184645 (+ 2720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:53.185327 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:53.185334 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:53.185336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:53.185476 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:53.185854 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:53.185857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:53.185859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:53.186211 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:53.186419 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:53.186700 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 04:57:53.186871 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 04:57:53.187954 20623 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:57:53.241467 20646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:53.240929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:44167 (local address 127.0.0.1:40406)
0925 04:57:53.241181 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:53.241285 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.129:44167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260925 04:57:53.242606 20646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:53.242213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:44167 (local address 127.0.0.1:40414)
0925 04:57:53.242425 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:53.242496 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.129:44167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260925 04:57:53.278488 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 20500
I20260925 04:57:53.295131 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 20139
I20260925 04:57:53.296473 20201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:57:53.410944 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 20139
I20260925 04:57:53.432662 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:39755
--webserver_interface=127.6.29.190
--webserver_port=36581
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:37825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:39755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:57:53.767124 20680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:57:53.767444 20680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:57:53.767537 20680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:57:53.775059 20680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:57:53.775175 20680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:57:53.775233 20680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:57:53.775295 20680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:57:53.775332 20680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:57:53.785351 20680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:37825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:39755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:39755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=36581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:57:53.786626 20680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:57:53.787945 20680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:57:53.811244 20686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:53.811305 20689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:53.811338 20687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:57:53.812160 20680 server_base.cc:1034] running on GCE node
I20260925 04:57:53.813037 20680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:57:53.814640 20680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:57:53.815960 20680 hybrid_clock.cc:648] HybridClock initialized: now 1790312273815901 us; error 33 us; skew 500 ppm
I20260925 04:57:53.816495 20680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:57:53 dist-test-slave-xzjr krb5kdc[20126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312273, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:57:53.820811 20680 init.cc:377] Logged in from keytab as kudu/127.6.29.190@KRBTEST.COM (short username kudu)
I20260925 04:57:53.822705 20680 webserver.cc:460] Webserver started at http://127.6.29.190:36581/ using document root <none> and password file <none>
I20260925 04:57:53.823504 20680 fs_manager.cc:359] Metadata directory not provided
I20260925 04:57:53.823665 20680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:57:53.830070 20680 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:57:53.833045 20696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:53.834436 20680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:57:53.834615 20680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "e28737f93785414ab258f58efc83f3c4"
format_stamp: "Formatted at 2026-09-25 04:57:50 on dist-test-slave-xzjr"
server_key: "9b6e52de35eadaa39c1d3a48011caaec"
server_key_iv: "bc6767f227e68d6cdee24806e1f192fb"
server_key_version: "encryptionkey@0"
I20260925 04:57:53.835732 20680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:57:53.847290 20680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:57:53.851552 20680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:57:53.851902 20680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:57:53.871618 20680 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:39755
I20260925 04:57:53.871627 20747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:39755 every 8 connection(s)
I20260925 04:57:53.873255 20680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:57:53.875640 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 20680
I20260925 04:57:53.876461 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.129:44167
--local_ip_for_outbound_sockets=127.6.29.129
--tserver_master_addrs=127.6.29.190:39755
--webserver_port=46169
--webserver_interface=127.6.29.129
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:37825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:57:53.880523 20748 sys_catalog.cc:263] Verifying existing consensus state
I20260925 04:57:53.883881 20748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4: Bootstrap starting.
I20260925 04:57:53.903472 20748 log.cc:824] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4: Log is configured to *not* fsync() on all Append() calls
I20260925 04:57:53.915367 20748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:57:53.916015 20748 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4: Bootstrap complete.
I20260925 04:57:53.925540 20748 raft_consensus.cc:348] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28737f93785414ab258f58efc83f3c4" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39755 } }
I20260925 04:57:53.926432 20748 raft_consensus.cc:729] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e28737f93785414ab258f58efc83f3c4, State: Initialized, Role: FOLLOWER
I20260925 04:57:53.927161 20748 consensus_queue.cc:260] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28737f93785414ab258f58efc83f3c4" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39755 } }
I20260925 04:57:53.927434 20748 raft_consensus.cc:388] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:57:53.927563 20748 raft_consensus.cc:482] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:57:53.927731 20748 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [term 1 FOLLOWER]: Advancing to term 2
I20260925 04:57:53.931674 20748 raft_consensus.cc:504] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28737f93785414ab258f58efc83f3c4" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39755 } }
I20260925 04:57:53.932256 20748 leader_election.cc:302] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e28737f93785414ab258f58efc83f3c4; no voters:
I20260925 04:57:53.932920 20748 leader_election.cc:288] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 04:57:53.933053 20752 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 04:57:53.934175 20752 raft_consensus.cc:686] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [term 2 LEADER]: Becoming Leader. State: Replica: e28737f93785414ab258f58efc83f3c4, State: Running, Role: LEADER
I20260925 04:57:53.934710 20752 consensus_queue.cc:237] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28737f93785414ab258f58efc83f3c4" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39755 } }
I20260925 04:57:53.935210 20748 sys_catalog.cc:564] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:57:53.937331 20754 sys_catalog.cc:455] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e28737f93785414ab258f58efc83f3c4. Latest consensus state: current_term: 2 leader_uuid: "e28737f93785414ab258f58efc83f3c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28737f93785414ab258f58efc83f3c4" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39755 } } }
I20260925 04:57:53.937724 20754 sys_catalog.cc:458] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [sys.catalog]: This master's current role is: LEADER
I20260925 04:57:53.937698 20753 sys_catalog.cc:455] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e28737f93785414ab258f58efc83f3c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28737f93785414ab258f58efc83f3c4" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39755 } } }
I20260925 04:57:53.938086 20753 sys_catalog.cc:458] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [sys.catalog]: This master's current role is: LEADER
I20260925 04:57:53.943290 20759 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:57:53.955868 20759 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:57:53.957859 20759 catalog_manager.cc:1259] Loaded cluster ID: 9763c0e1d58a45b6b6c6faaeb1946feb
I20260925 04:57:53.957991 20759 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:57:53.962097 20759 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 04:57:53.987495 20759 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 04:57:53.999015 20759 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:57:54.001302 20759 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4: Loaded TSK: 0
I20260925 04:57:54.003441 20759 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 04:57:54.257887 20750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:57:54.258261 20750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:57:54.258424 20750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:57:54.266081 20750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:57:54.266264 20750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:57:54.266438 20750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.129
I20260925 04:57:54.276777 20750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:37825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.129:44167
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.29.129
--webserver_port=46169
--tserver_master_addrs=127.6.29.190:39755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.129
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:57:54.278296 20750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:57:54.279657 20750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:57:54.294766 20776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:54.294799 20775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:54.295603 20778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:57:54.296236 20750 server_base.cc:1034] running on GCE node
I20260925 04:57:54.296901 20750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:57:54.297907 20750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:57:54.299151 20750 hybrid_clock.cc:648] HybridClock initialized: now 1790312274299084 us; error 60 us; skew 500 ppm
I20260925 04:57:54.299481 20750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:57:54 dist-test-slave-xzjr krb5kdc[20126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312274, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:57:54.309027 20750 init.cc:377] Logged in from keytab as kudu/127.6.29.129@KRBTEST.COM (short username kudu)
I20260925 04:57:54.310900 20750 webserver.cc:460] Webserver started at http://127.6.29.129:46169/ using document root <none> and password file <none>
I20260925 04:57:54.311671 20750 fs_manager.cc:359] Metadata directory not provided
I20260925 04:57:54.311873 20750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:57:54.317584 20750 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 04:57:54.320878 20785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:54.322422 20750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 04:57:54.322603 20750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/wal
uuid: "4a940a2e05694251b159ccbb2c4dda1c"
format_stamp: "Formatted at 2026-09-25 04:57:51 on dist-test-slave-xzjr"
server_key: "4afd8bcd8f07cdc64bd1749685dc7483"
server_key_iv: "69da25b82495cf436cc9170905d8fe96"
server_key_version: "encryptionkey@0"
I20260925 04:57:54.323864 20750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:57:54.341859 20750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:57:54.346114 20750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:57:54.346385 20750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:57:54.347481 20750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:57:54.349454 20792 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260925 04:57:54.354763 20792 ts_tablet_manager.cc:1852] T fdb56a2cbfff418f83d0ed1b33de7cb9 P 4a940a2e05694251b159ccbb2c4dda1c: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260925 04:57:54.355746 20792 ts_tablet_manager.cc:1905] T fdb56a2cbfff418f83d0ed1b33de7cb9 P 4a940a2e05694251b159ccbb2c4dda1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:57:54.363046 20792 ts_tablet_manager.cc:1918] T fdb56a2cbfff418f83d0ed1b33de7cb9 P 4a940a2e05694251b159ccbb2c4dda1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:57:54.363358 20792 ts_tablet_manager.cc:1939] T fdb56a2cbfff418f83d0ed1b33de7cb9 P 4a940a2e05694251b159ccbb2c4dda1c: Deleting consensus metadata
I20260925 04:57:54.363817 20750 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260925 04:57:54.363921 20750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.001s sys 0.000s
I20260925 04:57:54.364015 20750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:57:54.364069 20750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:54.397178 20750 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.129:44167
I20260925 04:57:54.397229 20898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.129:44167 every 8 connection(s)
I20260925 04:57:54.398566 20750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
I20260925 04:57:54.400281 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 20750
I20260925 04:57:54.401036 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.130:35163
--local_ip_for_outbound_sockets=127.6.29.130
--tserver_master_addrs=127.6.29.190:39755
--webserver_port=39135
--webserver_interface=127.6.29.130
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:37825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 04:57:54 dist-test-slave-xzjr krb5kdc[20126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312274, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:57:54.425686 20902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:54.401897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.129:51793)
0925 04:57:54.402910 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:54.402968 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 04:57:54.405870 (+ 2902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:57:54.407316 (+ 1446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:57:54.407329 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:57:54.408144 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:57:54.408826 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:54.408844 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:54.410752 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:54.410760 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:54.413083 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:54.413101 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:54.414076 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:54.414084 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:54.414392 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:54.415126 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:57:54.415160 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:57:54.417854 (+ 2694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:57:54.421315 (+ 3461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:54.421325 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:54.421340 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:54.421755 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:54.422186 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:54.422192 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:54.422196 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:54.422363 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:54.423081 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:57:54.423089 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:57:54.423437 (+ 348us) client_negotiation.cc:769] Sending connection context
0925 04:57:54.423809 (+ 372us) client_negotiation.cc:240] Negotiation successful
0925 04:57:54.424351 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":401,"threads_started":1}
I20260925 04:57:54.425684 20901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:54.401532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:51793 (local address 127.6.29.190:39755)
0925 04:57:54.403989 (+ 2457us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:54.404000 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:54.404046 (+ 46us) server_negotiation.cc:407] Connection header received
0925 04:57:54.406433 (+ 2387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:54.406447 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:54.406798 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:54.407095 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:54.409075 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:54.410581 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:54.413223 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:54.413843 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:54.413978 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:54.418147 (+ 4169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:54.418156 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:54.418172 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:54.418218 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:54.421176 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:54.421898 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:54.421904 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:54.421910 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:54.422075 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:54.422484 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:54.422489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:54.422496 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:54.422941 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:54.423140 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:54.424046 (+ 906us) server_negotiation.cc:299] Negotiation successful
0925 04:57:54.424421 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2262,"thread_start_us":2093,"threads_started":1}
I20260925 04:57:54.429013 20899 heartbeater.cc:347] Connected to a master server at 127.6.29.190:39755
I20260925 04:57:54.429356 20899 heartbeater.cc:464] Registering TS with master...
I20260925 04:57:54.430181 20899 heartbeater.cc:511] Master 127.6.29.190:39755 requested a full tablet report, sending...
I20260925 04:57:54.432650 20713 ts_manager.cc:194] Registered new tserver with Master: 4a940a2e05694251b159ccbb2c4dda1c (127.6.29.129:44167)
I20260925 04:57:54.434765 20713 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.129@KRBTEST.COM'} at 127.6.29.129:51793
W20260925 04:57:54.733225 20903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:57:54.733547 20903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:57:54.733644 20903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:57:54.745415 20903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:57:54.745527 20903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:57:54.745651 20903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.130
I20260925 04:57:54.755466 20903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:37825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.130:35163
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.29.130
--webserver_port=39135
--tserver_master_addrs=127.6.29.190:39755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.130
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:57:54.756776 20903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:57:54.758028 20903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:57:54.771286 20909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:54.771405 20908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:54.771381 20911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:57:54.772467 20903 server_base.cc:1034] running on GCE node
I20260925 04:57:54.773097 20903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:57:54.774026 20903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:57:54.775250 20903 hybrid_clock.cc:648] HybridClock initialized: now 1790312274775191 us; error 49 us; skew 500 ppm
I20260925 04:57:54.775576 20903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:57:54 dist-test-slave-xzjr krb5kdc[20126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312274, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:57:54.778594 20903 init.cc:377] Logged in from keytab as kudu/127.6.29.130@KRBTEST.COM (short username kudu)
I20260925 04:57:54.780016 20903 webserver.cc:460] Webserver started at http://127.6.29.130:39135/ using document root <none> and password file <none>
I20260925 04:57:54.780710 20903 fs_manager.cc:359] Metadata directory not provided
I20260925 04:57:54.780879 20903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:57:54.786157 20903 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260925 04:57:54.788866 20918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:54.790172 20903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 04:57:54.790344 20903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/wal
uuid: "e59fc77557484dd4a51d755847101947"
format_stamp: "Formatted at 2026-09-25 04:57:51 on dist-test-slave-xzjr"
server_key: "c38c87be4c7f1bb98772732cf809c71e"
server_key_iv: "c6ffd68bd8c9bf3d907b51b48104017a"
server_key_version: "encryptionkey@0"
I20260925 04:57:54.791321 20903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:57:54.803864 20903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:57:54.807732 20903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:57:54.808040 20903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:57:54.808960 20903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:57:54.810248 20903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:57:54.810371 20903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:54.810495 20903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:57:54.810592 20903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:54.848691 20903 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.130:35163
I20260925 04:57:54.848779 21030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.130:35163 every 8 connection(s)
I20260925 04:57:54.850234 20903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
I20260925 04:57:54.856860 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 20903
I20260925 04:57:54.857523 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.131:46361
--local_ip_for_outbound_sockets=127.6.29.131
--tserver_master_addrs=127.6.29.190:39755
--webserver_port=42127
--webserver_interface=127.6.29.131
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:37825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 04:57:54 dist-test-slave-xzjr krb5kdc[20126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312274, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:57:54.871031 20901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:54.852998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:36297 (local address 127.6.29.190:39755)
0925 04:57:54.853204 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:54.853210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:54.854404 (+ 1194us) server_negotiation.cc:407] Connection header received
0925 04:57:54.856428 (+ 2024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:54.856457 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:54.856547 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:54.856656 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:54.858326 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:54.859043 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:54.860665 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:54.861308 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:54.861488 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:54.865048 (+ 3560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:54.865052 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:54.865056 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:54.865090 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:54.867656 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:54.868438 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:54.868448 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:54.868451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:54.868579 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:54.869021 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:54.869031 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:54.869033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:54.869299 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:54.869460 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:54.870393 (+ 933us) server_negotiation.cc:299] Negotiation successful
0925 04:57:54.870635 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 04:57:54.872155 21033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:54.853498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.130:36297)
0925 04:57:54.854216 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:54.854269 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 04:57:54.856108 (+ 1839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:57:54.856906 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:57:54.856914 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:57:54.857490 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:57:54.858134 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:54.858152 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:54.859194 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:54.859204 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:54.860496 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:54.860507 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:54.861567 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:54.861575 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:54.861814 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:54.862538 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:57:54.862561 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:57:54.864821 (+ 2260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:57:54.867827 (+ 3006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:54.867836 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:54.867855 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:54.868301 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:54.868700 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:54.868705 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:54.868708 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:54.868894 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:54.869425 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:57:54.869433 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:57:54.869791 (+ 358us) client_negotiation.cc:769] Sending connection context
0925 04:57:54.870293 (+ 502us) client_negotiation.cc:240] Negotiation successful
0925 04:57:54.870917 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":288,"threads_started":1}
I20260925 04:57:54.873821 21031 heartbeater.cc:347] Connected to a master server at 127.6.29.190:39755
I20260925 04:57:54.874136 21031 heartbeater.cc:464] Registering TS with master...
I20260925 04:57:54.874936 21031 heartbeater.cc:511] Master 127.6.29.190:39755 requested a full tablet report, sending...
I20260925 04:57:54.876513 20713 ts_manager.cc:194] Registered new tserver with Master: e59fc77557484dd4a51d755847101947 (127.6.29.130:35163)
I20260925 04:57:54.877375 20713 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.130@KRBTEST.COM'} at 127.6.29.130:36297
W20260925 04:57:55.192731 21034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:57:55.193040 21034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:57:55.193215 21034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:57:55.200785 21034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:57:55.200897 21034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:57:55.201097 21034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.131
I20260925 04:57:55.210933 21034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:37825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.131:46361
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.29.131
--webserver_port=42127
--tserver_master_addrs=127.6.29.190:39755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.131
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:57:55.212375 21034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:57:55.213682 21034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:57:55.227232 21042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:55.227366 21039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:55.227366 21040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:57:55.228024 21034 server_base.cc:1034] running on GCE node
I20260925 04:57:55.228623 21034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:57:55.229628 21034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:57:55.230824 21034 hybrid_clock.cc:648] HybridClock initialized: now 1790312275230775 us; error 43 us; skew 500 ppm
I20260925 04:57:55.231158 21034 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:57:55 dist-test-slave-xzjr krb5kdc[20126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312275, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:57:55.234402 21034 init.cc:377] Logged in from keytab as kudu/127.6.29.131@KRBTEST.COM (short username kudu)
I20260925 04:57:55.235986 21034 webserver.cc:460] Webserver started at http://127.6.29.131:42127/ using document root <none> and password file <none>
I20260925 04:57:55.236685 21034 fs_manager.cc:359] Metadata directory not provided
I20260925 04:57:55.236833 21034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:57:55.242264 21034 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 04:57:55.245144 21049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:55.246500 21034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:57:55.246677 21034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/wal
uuid: "dff58da6112a4e8ba899f50028dd2b52"
format_stamp: "Formatted at 2026-09-25 04:57:52 on dist-test-slave-xzjr"
server_key: "b8ea3f81090e2492de138248abf6e460"
server_key_iv: "756f0ea0011f3845c45d63fb11b289ab"
server_key_version: "encryptionkey@0"
I20260925 04:57:55.247673 21034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:57:55.263989 21034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:57:55.268158 21034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:57:55.268450 21034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:57:55.269399 21034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:57:55.270682 21034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:57:55.270767 21034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:55.270854 21034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:57:55.270913 21034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:55.304064 21034 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.131:46361
I20260925 04:57:55.304136 21161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.131:46361 every 8 connection(s)
I20260925 04:57:55.305553 21034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
I20260925 04:57:55.310693 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 21034
Sep 25 04:57:55 dist-test-slave-xzjr krb5kdc[20126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312275, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:57:55.327520 21166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:55.311689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49108 (local address 127.6.29.190:39755)
0925 04:57:55.312995 (+ 1306us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:55.312999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:55.313012 (+ 13us) server_negotiation.cc:407] Connection header received
0925 04:57:55.313051 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:55.313054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:55.313100 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:55.313180 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:55.314841 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:55.315552 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:55.317433 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:55.318083 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:55.318188 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:55.321375 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:55.321379 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:55.321382 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:55.321412 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:55.324723 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:55.325508 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:55.325512 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:55.325514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:55.325622 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:55.326025 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:55.326029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:55.326032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:55.326319 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:55.326527 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:55.326858 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 04:57:55.327029 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1205,"spinlock_wait_cycles":3456,"thread_start_us":1093,"threads_started":1}
I20260925 04:57:55.327891 20901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:55.308189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:34035 (local address 127.6.29.190:39755)
0925 04:57:55.308396 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:55.308400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:55.309556 (+ 1156us) server_negotiation.cc:407] Connection header received
0925 04:57:55.311473 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:55.311481 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:55.311586 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:55.311708 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:55.313437 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:55.314138 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:55.316064 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:55.316713 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:55.316805 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:55.321492 (+ 4687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:55.321496 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:55.321499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:55.321541 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:55.324369 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:55.325238 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:55.325245 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:55.325249 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:55.325402 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:55.325898 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:55.325901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:55.325903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:55.326259 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:55.326401 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:55.327291 (+ 890us) server_negotiation.cc:299] Negotiation successful
0925 04:57:55.327508 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 04:57:55.329075 21164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:55.308669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.131:34035)
0925 04:57:55.309389 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:55.309429 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 04:57:55.311151 (+ 1722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:57:55.312063 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:57:55.312076 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:57:55.312647 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:57:55.313264 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:55.313289 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:55.314305 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:55.314311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:55.315894 (+ 1583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:55.315909 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:55.317141 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:55.317148 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:55.317472 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:55.318200 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:57:55.318242 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:57:55.321255 (+ 3013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:57:55.324531 (+ 3276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:55.324540 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:55.324557 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:55.325091 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:55.325545 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:55.325549 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:55.325553 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:55.325789 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:55.326409 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:57:55.326417 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:57:55.326861 (+ 444us) client_negotiation.cc:769] Sending connection context
0925 04:57:55.327187 (+ 326us) client_negotiation.cc:240] Negotiation successful
0925 04:57:55.327807 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":275,"threads_started":1}
I20260925 04:57:55.330485 21162 heartbeater.cc:347] Connected to a master server at 127.6.29.190:39755
I20260925 04:57:55.330801 21162 heartbeater.cc:464] Registering TS with master...
I20260925 04:57:55.331538 21162 heartbeater.cc:511] Master 127.6.29.190:39755 requested a full tablet report, sending...
I20260925 04:57:55.333009 20712 ts_manager.cc:194] Registered new tserver with Master: dff58da6112a4e8ba899f50028dd2b52 (127.6.29.131:46361)
I20260925 04:57:55.333900 20712 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.131@KRBTEST.COM'} at 127.6.29.131:34035
I20260925 04:57:55.342285 6262 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:57:55.349287 21166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:55.343158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49116 (local address 127.6.29.190:39755)
0925 04:57:55.343361 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:55.343366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:55.343818 (+ 452us) server_negotiation.cc:407] Connection header received
0925 04:57:55.344078 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:55.344083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:55.344232 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:55.344379 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:57:55.344780 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:55.345557 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:55.347074 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:55.347501 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:55.347626 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:55.348204 (+ 578us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:57:55.348721 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:57:55.348873 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:55.348929 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 04:57:55.349009 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 04:57:55.350201 20712 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:49116:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:57:55.357585 20712 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:57:55.394686 21172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:55.369957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.131:46361 (local address 127.0.0.1:44574)
0925 04:57:55.382595 (+ 12638us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:55.382647 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 04:57:55.382852 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:57:55.383981 (+ 1129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:57:55.383990 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:57:55.384033 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:57:55.384478 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:55.384496 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:55.387888 (+ 3392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:55.387897 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:55.391036 (+ 3139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:55.391059 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:55.393681 (+ 2622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:55.393688 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:55.393887 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:55.393941 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 04:57:55.394076 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 04:57:55.394266 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12350,"thread_start_us":12577,"threads_started":1}
I20260925 04:57:55.394681 21174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:55.370186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44574 (local address 127.6.29.131:46361)
0925 04:57:55.382503 (+ 12317us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:55.382513 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:55.383274 (+ 761us) server_negotiation.cc:407] Connection header received
0925 04:57:55.383410 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:55.383419 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:55.383700 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:55.383937 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:57:55.384714 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:55.386422 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:55.391274 (+ 4852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:55.393092 (+ 1818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:55.393294 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:55.393547 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:55.394151 (+ 604us) server_negotiation.cc:299] Negotiation successful
0925 04:57:55.394328 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12097,"thread_start_us":12813,"threads_started":1}
I20260925 04:57:55.396064 21173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:55.370008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:44167 (local address 127.0.0.1:40428)
0925 04:57:55.382973 (+ 12965us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:55.382997 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 04:57:55.383147 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:57:55.384577 (+ 1430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:57:55.384584 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:57:55.384613 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:57:55.385012 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:55.385025 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:55.387001 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:55.387007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:55.393048 (+ 6041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:55.393064 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:55.394951 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:55.394958 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:55.395237 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:55.395336 (+ 99us) client_negotiation.cc:769] Sending connection context
0925 04:57:55.395484 (+ 148us) client_negotiation.cc:240] Negotiation successful
0925 04:57:55.395620 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12793,"mutex_wait_us":93,"thread_start_us":13243,"threads_started":1}
I20260925 04:57:55.396870 21175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:55.370479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40428 (local address 127.6.29.129:44167)
0925 04:57:55.383953 (+ 13474us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:55.383962 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:55.383990 (+ 28us) server_negotiation.cc:407] Connection header received
0925 04:57:55.384082 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:55.384091 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:55.384341 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:55.385219 (+ 878us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:57:55.385438 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:55.386844 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:55.393253 (+ 6409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:55.394691 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:55.395326 (+ 635us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:55.395532 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:55.395598 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 04:57:55.395765 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13295,"thread_start_us":13132,"threads_started":1}
I20260925 04:57:55.398911 20834 tablet_service.cc:1467] Processing CreateTablet for tablet 23fec7a7429b4884ac342d0319e6c0ca (DEFAULT_TABLE table=test-table [id=a1ab717e395c4464b1f8448832733da0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:57:55.399278 21097 tablet_service.cc:1467] Processing CreateTablet for tablet 23fec7a7429b4884ac342d0319e6c0ca (DEFAULT_TABLE table=test-table [id=a1ab717e395c4464b1f8448832733da0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:57:55.400393 20834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23fec7a7429b4884ac342d0319e6c0ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:57:55.400533 21177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:55.384462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:35163 (local address 127.0.0.1:43852)
0925 04:57:55.391613 (+ 7151us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:55.391628 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:57:55.391807 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:57:55.392391 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:57:55.392395 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:57:55.392420 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:57:55.392923 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:55.392936 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:55.395147 (+ 2211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:55.395153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:55.397422 (+ 2269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:55.397438 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:55.399641 (+ 2203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:55.399648 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:55.399847 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:55.399878 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 04:57:55.399968 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 04:57:55.400099 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7033,"mutex_wait_us":3,"thread_start_us":6856,"threads_started":1}
I20260925 04:57:55.400686 21097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23fec7a7429b4884ac342d0319e6c0ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:57:55.401397 21176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:55.383700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43852 (local address 127.6.29.130:35163)
0925 04:57:55.387677 (+ 3977us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:55.387689 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:55.391743 (+ 4054us) server_negotiation.cc:407] Connection header received
0925 04:57:55.391998 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:55.392004 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:55.392204 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:55.392404 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:57:55.393144 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:55.394601 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:55.397627 (+ 3026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:55.399148 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:55.399291 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:55.399492 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:55.400869 (+ 1377us) server_negotiation.cc:299] Negotiation successful
0925 04:57:55.401049 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3811,"thread_start_us":3689,"threads_started":1}
I20260925 04:57:55.405175 20966 tablet_service.cc:1467] Processing CreateTablet for tablet 23fec7a7429b4884ac342d0319e6c0ca (DEFAULT_TABLE table=test-table [id=a1ab717e395c4464b1f8448832733da0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:57:55.406641 20966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23fec7a7429b4884ac342d0319e6c0ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:57:55.410807 21178 tablet_bootstrap.cc:492] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c: Bootstrap starting.
I20260925 04:57:55.414582 21178 tablet_bootstrap.cc:654] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c: Neither blocks nor log segments found. Creating new log.
I20260925 04:57:55.415982 21178 log.cc:824] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c: Log is configured to *not* fsync() on all Append() calls
I20260925 04:57:55.416604 21179 tablet_bootstrap.cc:492] T 23fec7a7429b4884ac342d0319e6c0ca P dff58da6112a4e8ba899f50028dd2b52: Bootstrap starting.
I20260925 04:57:55.419232 21178 tablet_bootstrap.cc:492] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c: No bootstrap required, opened a new log
I20260925 04:57:55.419559 21178 ts_tablet_manager.cc:1397] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 04:57:55.420157 21179 tablet_bootstrap.cc:654] T 23fec7a7429b4884ac342d0319e6c0ca P dff58da6112a4e8ba899f50028dd2b52: Neither blocks nor log segments found. Creating new log.
I20260925 04:57:55.421813 21179 log.cc:824] T 23fec7a7429b4884ac342d0319e6c0ca P dff58da6112a4e8ba899f50028dd2b52: Log is configured to *not* fsync() on all Append() calls
I20260925 04:57:55.423243 21181 tablet_bootstrap.cc:492] T 23fec7a7429b4884ac342d0319e6c0ca P e59fc77557484dd4a51d755847101947: Bootstrap starting.
I20260925 04:57:55.425074 21179 tablet_bootstrap.cc:492] T 23fec7a7429b4884ac342d0319e6c0ca P dff58da6112a4e8ba899f50028dd2b52: No bootstrap required, opened a new log
I20260925 04:57:55.425354 21179 ts_tablet_manager.cc:1397] T 23fec7a7429b4884ac342d0319e6c0ca P dff58da6112a4e8ba899f50028dd2b52: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260925 04:57:55.426362 21181 tablet_bootstrap.cc:654] T 23fec7a7429b4884ac342d0319e6c0ca P e59fc77557484dd4a51d755847101947: Neither blocks nor log segments found. Creating new log.
I20260925 04:57:55.427470 21181 log.cc:824] T 23fec7a7429b4884ac342d0319e6c0ca P e59fc77557484dd4a51d755847101947: Log is configured to *not* fsync() on all Append() calls
I20260925 04:57:55.430130 21181 tablet_bootstrap.cc:492] T 23fec7a7429b4884ac342d0319e6c0ca P e59fc77557484dd4a51d755847101947: No bootstrap required, opened a new log
I20260925 04:57:55.430382 21181 ts_tablet_manager.cc:1397] T 23fec7a7429b4884ac342d0319e6c0ca P e59fc77557484dd4a51d755847101947: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260925 04:57:55.431052 21178 raft_consensus.cc:348] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff58da6112a4e8ba899f50028dd2b52" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 46361 } } peers { permanent_uuid: "4a940a2e05694251b159ccbb2c4dda1c" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 44167 } } peers { permanent_uuid: "e59fc77557484dd4a51d755847101947" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35163 } }
I20260925 04:57:55.431515 21178 raft_consensus.cc:374] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:57:55.431613 21178 raft_consensus.cc:729] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a940a2e05694251b159ccbb2c4dda1c, State: Initialized, Role: FOLLOWER
I20260925 04:57:55.432315 21178 consensus_queue.cc:260] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff58da6112a4e8ba899f50028dd2b52" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 46361 } } peers { permanent_uuid: "4a940a2e05694251b159ccbb2c4dda1c" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 44167 } } peers { permanent_uuid: "e59fc77557484dd4a51d755847101947" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35163 } }
I20260925 04:57:55.434752 20899 heartbeater.cc:503] Master 127.6.29.190:39755 was elected leader, sending a full tablet report...
I20260925 04:57:55.435498 21178 ts_tablet_manager.cc:1428] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260925 04:57:55.436329 21179 raft_consensus.cc:348] T 23fec7a7429b4884ac342d0319e6c0ca P dff58da6112a4e8ba899f50028dd2b52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff58da6112a4e8ba899f50028dd2b52" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 46361 } } peers { permanent_uuid: "4a940a2e05694251b159ccbb2c4dda1c" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 44167 } } peers { permanent_uuid: "e59fc77557484dd4a51d755847101947" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35163 } }
I20260925 04:57:55.436703 21179 raft_consensus.cc:374] T 23fec7a7429b4884ac342d0319e6c0ca P dff58da6112a4e8ba899f50028dd2b52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:57:55.436831 21179 raft_consensus.cc:729] T 23fec7a7429b4884ac342d0319e6c0ca P dff58da6112a4e8ba899f50028dd2b52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff58da6112a4e8ba899f50028dd2b52, State: Initialized, Role: FOLLOWER
I20260925 04:57:55.437435 21179 consensus_queue.cc:260] T 23fec7a7429b4884ac342d0319e6c0ca P dff58da6112a4e8ba899f50028dd2b52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff58da6112a4e8ba899f50028dd2b52" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 46361 } } peers { permanent_uuid: "4a940a2e05694251b159ccbb2c4dda1c" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 44167 } } peers { permanent_uuid: "e59fc77557484dd4a51d755847101947" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35163 } }
I20260925 04:57:55.438510 21162 heartbeater.cc:503] Master 127.6.29.190:39755 was elected leader, sending a full tablet report...
I20260925 04:57:55.439203 21179 ts_tablet_manager.cc:1428] T 23fec7a7429b4884ac342d0319e6c0ca P dff58da6112a4e8ba899f50028dd2b52: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260925 04:57:55.440025 21181 raft_consensus.cc:348] T 23fec7a7429b4884ac342d0319e6c0ca P e59fc77557484dd4a51d755847101947 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff58da6112a4e8ba899f50028dd2b52" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 46361 } } peers { permanent_uuid: "4a940a2e05694251b159ccbb2c4dda1c" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 44167 } } peers { permanent_uuid: "e59fc77557484dd4a51d755847101947" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35163 } }
I20260925 04:57:55.440526 21181 raft_consensus.cc:374] T 23fec7a7429b4884ac342d0319e6c0ca P e59fc77557484dd4a51d755847101947 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:57:55.440632 21181 raft_consensus.cc:729] T 23fec7a7429b4884ac342d0319e6c0ca P e59fc77557484dd4a51d755847101947 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59fc77557484dd4a51d755847101947, State: Initialized, Role: FOLLOWER
I20260925 04:57:55.441256 21181 consensus_queue.cc:260] T 23fec7a7429b4884ac342d0319e6c0ca P e59fc77557484dd4a51d755847101947 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff58da6112a4e8ba899f50028dd2b52" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 46361 } } peers { permanent_uuid: "4a940a2e05694251b159ccbb2c4dda1c" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 44167 } } peers { permanent_uuid: "e59fc77557484dd4a51d755847101947" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35163 } }
I20260925 04:57:55.443724 21031 heartbeater.cc:503] Master 127.6.29.190:39755 was elected leader, sending a full tablet report...
I20260925 04:57:55.444528 21181 ts_tablet_manager.cc:1428] T 23fec7a7429b4884ac342d0319e6c0ca P e59fc77557484dd4a51d755847101947: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
W20260925 04:57:55.557291 21163 tablet.cc:2369] T 23fec7a7429b4884ac342d0319e6c0ca P dff58da6112a4e8ba899f50028dd2b52: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:57:55.602375 21032 tablet.cc:2369] T 23fec7a7429b4884ac342d0319e6c0ca P e59fc77557484dd4a51d755847101947: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:57:55.628142 21186 raft_consensus.cc:482] T 23fec7a7429b4884ac342d0319e6c0ca P e59fc77557484dd4a51d755847101947 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:57:55.628494 21186 raft_consensus.cc:504] T 23fec7a7429b4884ac342d0319e6c0ca P e59fc77557484dd4a51d755847101947 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff58da6112a4e8ba899f50028dd2b52" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 46361 } } peers { permanent_uuid: "4a940a2e05694251b159ccbb2c4dda1c" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 44167 } } peers { permanent_uuid: "e59fc77557484dd4a51d755847101947" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35163 } }
I20260925 04:57:55.630095 21186 leader_election.cc:288] T 23fec7a7429b4884ac342d0319e6c0ca P e59fc77557484dd4a51d755847101947 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff58da6112a4e8ba899f50028dd2b52 (127.6.29.131:46361), 4a940a2e05694251b159ccbb2c4dda1c (127.6.29.129:44167)
I20260925 04:57:55.639467 21174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:55.630657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:51217 (local address 127.6.29.131:46361)
0925 04:57:55.630835 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:55.630842 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:55.631569 (+ 727us) server_negotiation.cc:407] Connection header received
0925 04:57:55.631804 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:55.631819 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:55.631895 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:55.632001 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:57:55.632788 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:55.633899 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:55.637084 (+ 3185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:55.638348 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:55.638511 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:55.638583 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:55.639055 (+ 472us) server_negotiation.cc:299] Negotiation successful
0925 04:57:55.639186 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 04:57:55.639489 21175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:55.630555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:41087 (local address 127.6.29.129:44167)
0925 04:57:55.630782 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:55.630789 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:55.631619 (+ 830us) server_negotiation.cc:407] Connection header received
0925 04:57:55.631804 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:55.631810 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:55.631903 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:55.632001 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:57:55.632788 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:55.633920 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:55.637084 (+ 3164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:55.638348 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:55.638506 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:55.638583 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:55.639055 (+ 472us) server_negotiation.cc:299] Negotiation successful
0925 04:57:55.639202 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 04:57:55.639799 21187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:55.630544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:44167 (local address 127.6.29.130:41087)
0925 04:57:55.631467 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:55.631482 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:57:55.631619 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:57:55.632030 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:57:55.632035 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:57:55.632065 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:57:55.632628 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:55.632643 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:55.634041 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:55.634048 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:55.636924 (+ 2876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:55.636941 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:55.638612 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:55.638619 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:55.638880 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:55.638921 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 04:57:55.639018 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 04:57:55.639278 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"mutex_wait_us":341,"thread_start_us":428,"threads_started":1}
I20260925 04:57:55.640187 21188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:55.630392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.131:46361 (local address 127.6.29.130:51217)
0925 04:57:55.631433 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:55.631458 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 04:57:55.631649 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:57:55.632030 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:57:55.632035 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:57:55.632081 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:57:55.632638 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:55.632646 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:55.634051 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:55.634055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:55.636924 (+ 2869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:55.636941 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:55.638617 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:55.638622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:55.638874 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:55.638928 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 04:57:55.639018 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 04:57:55.639278 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":874,"mutex_wait_us":301,"spinlock_wait_cycles":51456,"thread_start_us":686,"threads_started":1}
I20260925 04:57:55.640885 20854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23fec7a7429b4884ac342d0319e6c0ca" candidate_uuid: "e59fc77557484dd4a51d755847101947" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a940a2e05694251b159ccbb2c4dda1c" is_pre_election: true
I20260925 04:57:55.641227 21117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23fec7a7429b4884ac342d0319e6c0ca" candidate_uuid: "e59fc77557484dd4a51d755847101947" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff58da6112a4e8ba899f50028dd2b52" is_pre_election: true
I20260925 04:57:55.641507 20854 raft_consensus.cc:2454] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e59fc77557484dd4a51d755847101947 in term 0.
I20260925 04:57:55.641737 21117 raft_consensus.cc:2454] T 23fec7a7429b4884ac342d0319e6c0ca P dff58da6112a4e8ba899f50028dd2b52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e59fc77557484dd4a51d755847101947 in term 0.
I20260925 04:57:55.642300 20922 leader_election.cc:302] T 23fec7a7429b4884ac342d0319e6c0ca P e59fc77557484dd4a51d755847101947 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4a940a2e05694251b159ccbb2c4dda1c, dff58da6112a4e8ba899f50028dd2b52, e59fc77557484dd4a51d755847101947; no voters:
I20260925 04:57:55.642738 21186 raft_consensus.cc:2781] T 23fec7a7429b4884ac342d0319e6c0ca P e59fc77557484dd4a51d755847101947 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:57:55.642889 21186 raft_consensus.cc:482] T 23fec7a7429b4884ac342d0319e6c0ca P e59fc77557484dd4a51d755847101947 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:57:55.643003 21186 raft_consensus.cc:3037] T 23fec7a7429b4884ac342d0319e6c0ca P e59fc77557484dd4a51d755847101947 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:57:55.645243 21186 raft_consensus.cc:504] T 23fec7a7429b4884ac342d0319e6c0ca P e59fc77557484dd4a51d755847101947 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff58da6112a4e8ba899f50028dd2b52" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 46361 } } peers { permanent_uuid: "4a940a2e05694251b159ccbb2c4dda1c" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 44167 } } peers { permanent_uuid: "e59fc77557484dd4a51d755847101947" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35163 } }
I20260925 04:57:55.645808 21186 leader_election.cc:288] T 23fec7a7429b4884ac342d0319e6c0ca P e59fc77557484dd4a51d755847101947 [CANDIDATE]: Term 1 election: Requested vote from peers dff58da6112a4e8ba899f50028dd2b52 (127.6.29.131:46361), 4a940a2e05694251b159ccbb2c4dda1c (127.6.29.129:44167)
I20260925 04:57:55.646255 20854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23fec7a7429b4884ac342d0319e6c0ca" candidate_uuid: "e59fc77557484dd4a51d755847101947" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a940a2e05694251b159ccbb2c4dda1c"
I20260925 04:57:55.646266 21117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23fec7a7429b4884ac342d0319e6c0ca" candidate_uuid: "e59fc77557484dd4a51d755847101947" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff58da6112a4e8ba899f50028dd2b52"
I20260925 04:57:55.646456 20854 raft_consensus.cc:3037] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:57:55.646466 21117 raft_consensus.cc:3037] T 23fec7a7429b4884ac342d0319e6c0ca P dff58da6112a4e8ba899f50028dd2b52 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:57:55.648664 21117 raft_consensus.cc:2454] T 23fec7a7429b4884ac342d0319e6c0ca P dff58da6112a4e8ba899f50028dd2b52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59fc77557484dd4a51d755847101947 in term 1.
I20260925 04:57:55.648726 20854 raft_consensus.cc:2454] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59fc77557484dd4a51d755847101947 in term 1.
I20260925 04:57:55.649091 20921 leader_election.cc:302] T 23fec7a7429b4884ac342d0319e6c0ca P e59fc77557484dd4a51d755847101947 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff58da6112a4e8ba899f50028dd2b52, e59fc77557484dd4a51d755847101947; no voters:
I20260925 04:57:55.649416 21186 raft_consensus.cc:2781] T 23fec7a7429b4884ac342d0319e6c0ca P e59fc77557484dd4a51d755847101947 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:57:55.649871 21186 raft_consensus.cc:686] T 23fec7a7429b4884ac342d0319e6c0ca P e59fc77557484dd4a51d755847101947 [term 1 LEADER]: Becoming Leader. State: Replica: e59fc77557484dd4a51d755847101947, State: Running, Role: LEADER
I20260925 04:57:55.650360 21186 consensus_queue.cc:237] T 23fec7a7429b4884ac342d0319e6c0ca P e59fc77557484dd4a51d755847101947 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff58da6112a4e8ba899f50028dd2b52" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 46361 } } peers { permanent_uuid: "4a940a2e05694251b159ccbb2c4dda1c" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 44167 } } peers { permanent_uuid: "e59fc77557484dd4a51d755847101947" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35163 } }
W20260925 04:57:55.650992 20900 tablet.cc:2369] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:57:55.655476 20713 catalog_manager.cc:5495] T 23fec7a7429b4884ac342d0319e6c0ca P e59fc77557484dd4a51d755847101947 reported cstate change: term changed from 0 to 1, leader changed from <none> to e59fc77557484dd4a51d755847101947 (127.6.29.130). New cstate: current_term: 1 leader_uuid: "e59fc77557484dd4a51d755847101947" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff58da6112a4e8ba899f50028dd2b52" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 46361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a940a2e05694251b159ccbb2c4dda1c" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 44167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59fc77557484dd4a51d755847101947" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35163 } health_report { overall_health: HEALTHY } } }
I20260925 04:57:55.675240 21176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:55.670248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43854 (local address 127.6.29.130:35163)
0925 04:57:55.670440 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:55.670444 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:55.670461 (+ 17us) server_negotiation.cc:407] Connection header received
0925 04:57:55.670542 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:55.670545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:55.670642 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:55.670763 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:57:55.671094 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:55.671898 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:55.673376 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:55.673763 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:55.673854 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:55.674304 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:57:55.674793 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:57:55.674923 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:55.674970 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 04:57:55.675036 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 04:57:55.683923 21117 raft_consensus.cc:1261] T 23fec7a7429b4884ac342d0319e6c0ca P dff58da6112a4e8ba899f50028dd2b52 [term 1 FOLLOWER]: Refusing update from remote peer e59fc77557484dd4a51d755847101947: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:57:55.683909 20854 raft_consensus.cc:1261] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [term 1 FOLLOWER]: Refusing update from remote peer e59fc77557484dd4a51d755847101947: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:57:55.685052 21189 consensus_queue.cc:1035] T 23fec7a7429b4884ac342d0319e6c0ca P e59fc77557484dd4a51d755847101947 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff58da6112a4e8ba899f50028dd2b52" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 46361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:57:55.685433 21186 consensus_queue.cc:1035] T 23fec7a7429b4884ac342d0319e6c0ca P e59fc77557484dd4a51d755847101947 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a940a2e05694251b159ccbb2c4dda1c" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 44167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:57:55.697395 21192 mvcc.cc:205] Tried to move back new op lower bound from 7333119081189728256 to 7333119081068691456. Current Snapshot: MvccSnapshot[applied={T|T < 7333119081189728256}]
I20260925 04:57:55.712852 20713 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:49116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:57:55.713130 20713 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:49116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:57:55.716547 20713 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4: Sending DeleteTablet for 3 replicas of tablet 23fec7a7429b4884ac342d0319e6c0ca
I20260925 04:57:55.717777 20834 tablet_service.cc:1514] Processing DeleteTablet for tablet 23fec7a7429b4884ac342d0319e6c0ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:57:55 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:40428
I20260925 04:57:55.718120 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 20750
I20260925 04:57:55.718283 21097 tablet_service.cc:1514] Processing DeleteTablet for tablet 23fec7a7429b4884ac342d0319e6c0ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:57:55 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:44574
I20260925 04:57:55.718976 20966 tablet_service.cc:1514] Processing DeleteTablet for tablet 23fec7a7429b4884ac342d0319e6c0ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:57:55 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:43852
I20260925 04:57:55.738524 21205 tablet_replica.cc:330] T 23fec7a7429b4884ac342d0319e6c0ca P e59fc77557484dd4a51d755847101947: stopping tablet replica
I20260925 04:57:55.741634 21175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:55.720179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40430 (local address 127.6.29.129:44167)
0925 04:57:55.720404 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:55.720411 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:55.720434 (+ 23us) server_negotiation.cc:407] Connection header received
0925 04:57:55.720506 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:55.720514 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:55.720647 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:55.720775 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:55.722725 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:55.724349 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:55.726515 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:55.727381 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:55.727533 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:55.731332 (+ 3799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:55.731355 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:55.731381 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:55.731438 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:55.736584 (+ 5146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:55.737537 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:55.737550 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:55.737557 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:55.737766 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:55.738750 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:55.738758 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:55.738764 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:55.739346 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:55.739807 (+ 461us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:55.740726 (+ 919us) server_negotiation.cc:299] Negotiation successful
0925 04:57:55.741050 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 04:57:55.742714 21205 raft_consensus.cc:2229] T 23fec7a7429b4884ac342d0319e6c0ca P e59fc77557484dd4a51d755847101947 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:57:55.742985 20894 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:57:55.743664 21205 raft_consensus.cc:2258] T 23fec7a7429b4884ac342d0319e6c0ca P e59fc77557484dd4a51d755847101947 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:57:55.746802 21205 ts_tablet_manager.cc:1905] T 23fec7a7429b4884ac342d0319e6c0ca P e59fc77557484dd4a51d755847101947: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:57:55.751197 21204 tablet_replica.cc:330] T 23fec7a7429b4884ac342d0319e6c0ca P dff58da6112a4e8ba899f50028dd2b52: stopping tablet replica
I20260925 04:57:55.751818 21204 raft_consensus.cc:2229] T 23fec7a7429b4884ac342d0319e6c0ca P dff58da6112a4e8ba899f50028dd2b52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:57:55.752501 21204 raft_consensus.cc:2258] T 23fec7a7429b4884ac342d0319e6c0ca P dff58da6112a4e8ba899f50028dd2b52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:57:55.755224 21204 ts_tablet_manager.cc:1905] T 23fec7a7429b4884ac342d0319e6c0ca P dff58da6112a4e8ba899f50028dd2b52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:57:55.759382 21205 ts_tablet_manager.cc:1918] T 23fec7a7429b4884ac342d0319e6c0ca P e59fc77557484dd4a51d755847101947: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:57:55.759677 21205 log.cc:1196] T 23fec7a7429b4884ac342d0319e6c0ca P e59fc77557484dd4a51d755847101947: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/wal/wals/23fec7a7429b4884ac342d0319e6c0ca
I20260925 04:57:55.760533 21205 ts_tablet_manager.cc:1939] T 23fec7a7429b4884ac342d0319e6c0ca P e59fc77557484dd4a51d755847101947: Deleting consensus metadata
I20260925 04:57:55.763001 20699 catalog_manager.cc:4841] TS e59fc77557484dd4a51d755847101947 (127.6.29.130:35163): tablet 23fec7a7429b4884ac342d0319e6c0ca (table test-table [id=a1ab717e395c4464b1f8448832733da0]) successfully deleted
I20260925 04:57:55.764277 21204 ts_tablet_manager.cc:1918] T 23fec7a7429b4884ac342d0319e6c0ca P dff58da6112a4e8ba899f50028dd2b52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:57:55.764624 21204 log.cc:1196] T 23fec7a7429b4884ac342d0319e6c0ca P dff58da6112a4e8ba899f50028dd2b52: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/wal/wals/23fec7a7429b4884ac342d0319e6c0ca
I20260925 04:57:55.765304 21204 ts_tablet_manager.cc:1939] T 23fec7a7429b4884ac342d0319e6c0ca P dff58da6112a4e8ba899f50028dd2b52: Deleting consensus metadata
I20260925 04:57:55.767067 20699 catalog_manager.cc:4841] TS dff58da6112a4e8ba899f50028dd2b52 (127.6.29.131:46361): tablet 23fec7a7429b4884ac342d0319e6c0ca (table test-table [id=a1ab717e395c4464b1f8448832733da0]) successfully deleted
I20260925 04:57:55.878288 21203 tablet_replica.cc:330] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c: stopping tablet replica
I20260925 04:57:55.878777 21203 raft_consensus.cc:2229] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:57:55.879300 21203 raft_consensus.cc:2258] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:57:55.881604 21203 ts_tablet_manager.cc:1905] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:57:55.882709 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 20750
W20260925 04:57:55.906657 20700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.29.129:44167 (error 108)
I20260925 04:57:55.907449 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 20903
I20260925 04:57:55.911990 21208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:55.911038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:44167 (local address 127.0.0.1:40446)
0925 04:57:55.911497 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:55.911631 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.129:44167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":227,"threads_started":1}
W20260925 04:57:55.912263 20700 catalog_manager.cc:4584] TS 4a940a2e05694251b159ccbb2c4dda1c (127.6.29.129:44167): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 23fec7a7429b4884ac342d0319e6c0ca: Network error: Client connection negotiation failed: client connection to 127.6.29.129:44167: connect: Connection refused (error 111)
I20260925 04:57:55.922547 21176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:55.908683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43866 (local address 127.6.29.130:35163)
0925 04:57:55.908882 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:55.908888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:55.908909 (+ 21us) server_negotiation.cc:407] Connection header received
0925 04:57:55.908974 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:55.908980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:55.909091 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:55.909202 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:55.911159 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:55.912260 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:55.913537 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:55.913977 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:55.914099 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:55.916733 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:55.916749 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:55.916763 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:55.916812 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:55.919744 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:55.920502 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:55.920510 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:55.920512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:55.920649 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:55.921023 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:55.921029 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:55.921033 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:55.921388 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:55.921708 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:55.921967 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 04:57:55.922182 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 04:57:55.923292 21026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:57:55.966194 21208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:55.965644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:44167 (local address 127.0.0.1:40462)
0925 04:57:55.965890 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:55.966012 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.129:44167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260925 04:57:55.967255 21208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:55.966847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:44167 (local address 127.0.0.1:40468)
0925 04:57:55.967014 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:55.967101 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.129:44167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 04:57:56.027953 21208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:56.027323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:44167 (local address 127.0.0.1:40482)
0925 04:57:56.027578 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:56.027698 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.129:44167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 04:57:56.029276 21208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:56.028838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:44167 (local address 127.0.0.1:40494)
0925 04:57:56.029004 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:56.029097 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.129:44167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 04:57:56.039572 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 20903
I20260925 04:57:56.058763 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 21034
I20260925 04:57:56.072029 21174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:56.059474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44588 (local address 127.6.29.131:46361)
0925 04:57:56.059673 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:56.059680 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:56.059783 (+ 103us) server_negotiation.cc:407] Connection header received
0925 04:57:56.060009 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:56.060013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:56.060077 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:56.060177 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:56.061491 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:56.062200 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:56.063527 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:56.063965 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:56.064087 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:56.066546 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:56.066559 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:56.066574 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:56.066609 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:56.069158 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:56.069843 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:56.069851 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:56.069853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:56.070043 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:56.070411 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:56.070417 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:56.070423 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:56.070749 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:56.071046 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:56.071341 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 04:57:56.071552 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 04:57:56.072645 21157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:57:56.109938 21208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:56.109464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:44167 (local address 127.0.0.1:40502)
0925 04:57:56.109674 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:56.109765 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.129:44167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 04:57:56.111040 21208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:56.110621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:44167 (local address 127.0.0.1:40512)
0925 04:57:56.110825 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:56.110897 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.129:44167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260925 04:57:56.164539 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 21034
I20260925 04:57:56.181427 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 20680
I20260925 04:57:56.182621 20743 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:57:56.299302 21208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:56.298706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:44167 (local address 127.0.0.1:40514)
0925 04:57:56.298934 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:56.299086 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.129:44167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 04:57:56.299546 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 20680
I20260925 04:57:56.321621 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:39755
--webserver_interface=127.6.29.190
--webserver_port=36581
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:37825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:39755
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:57:56.649873 21212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:57:56.650195 21212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:57:56.650288 21212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:57:56.657783 21212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:57:56.657910 21212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:57:56.657958 21212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:57:56.658008 21212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:57:56.658052 21212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:57:56.668779 21212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:37825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:39755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:39755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=36581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:57:56.670125 21212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:57:56.671962 21212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:57:56.683831 21220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:56.683917 21218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:56.683812 21217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:57:56.685273 21212 server_base.cc:1034] running on GCE node
I20260925 04:57:56.686050 21212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:57:56.687456 21212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:57:56.688681 21212 hybrid_clock.cc:648] HybridClock initialized: now 1790312276688634 us; error 56 us; skew 500 ppm
I20260925 04:57:56.689028 21212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:57:56 dist-test-slave-xzjr krb5kdc[20126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312276, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:57:56.692302 21212 init.cc:377] Logged in from keytab as kudu/127.6.29.190@KRBTEST.COM (short username kudu)
I20260925 04:57:56.693867 21212 webserver.cc:460] Webserver started at http://127.6.29.190:36581/ using document root <none> and password file <none>
I20260925 04:57:56.694555 21212 fs_manager.cc:359] Metadata directory not provided
I20260925 04:57:56.694681 21212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:57:56.700335 21212 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 04:57:56.703328 21227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:56.704779 21212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:57:56.705063 21212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "e28737f93785414ab258f58efc83f3c4"
format_stamp: "Formatted at 2026-09-25 04:57:50 on dist-test-slave-xzjr"
server_key: "9b6e52de35eadaa39c1d3a48011caaec"
server_key_iv: "bc6767f227e68d6cdee24806e1f192fb"
server_key_version: "encryptionkey@0"
I20260925 04:57:56.706073 21212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:57:56.729729 21212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:57:56.734211 21212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:57:56.734498 21212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:57:56.753505 21212 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:39755
I20260925 04:57:56.753539 21278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:39755 every 8 connection(s)
I20260925 04:57:56.755023 21212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:57:56.761754 21279 sys_catalog.cc:263] Verifying existing consensus state
I20260925 04:57:56.764595 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 21212
I20260925 04:57:56.765098 21279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4: Bootstrap starting.
I20260925 04:57:56.765431 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.129:44167
--local_ip_for_outbound_sockets=127.6.29.129
--tserver_master_addrs=127.6.29.190:39755
--webserver_port=46169
--webserver_interface=127.6.29.129
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:37825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:57:56.786339 21279 log.cc:824] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4: Log is configured to *not* fsync() on all Append() calls
I20260925 04:57:56.802592 21279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:57:56.803161 21279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4: Bootstrap complete.
I20260925 04:57:56.812637 21279 raft_consensus.cc:348] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28737f93785414ab258f58efc83f3c4" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39755 } }
I20260925 04:57:56.813503 21279 raft_consensus.cc:729] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e28737f93785414ab258f58efc83f3c4, State: Initialized, Role: FOLLOWER
I20260925 04:57:56.814131 21279 consensus_queue.cc:260] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28737f93785414ab258f58efc83f3c4" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39755 } }
I20260925 04:57:56.814368 21279 raft_consensus.cc:388] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:57:56.814486 21279 raft_consensus.cc:482] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:57:56.814667 21279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [term 2 FOLLOWER]: Advancing to term 3
I20260925 04:57:56.818567 21279 raft_consensus.cc:504] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28737f93785414ab258f58efc83f3c4" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39755 } }
I20260925 04:57:56.819072 21279 leader_election.cc:302] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e28737f93785414ab258f58efc83f3c4; no voters:
I20260925 04:57:56.819753 21279 leader_election.cc:288] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 04:57:56.819921 21283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 04:57:56.821087 21283 raft_consensus.cc:686] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [term 3 LEADER]: Becoming Leader. State: Replica: e28737f93785414ab258f58efc83f3c4, State: Running, Role: LEADER
I20260925 04:57:56.821693 21283 consensus_queue.cc:237] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28737f93785414ab258f58efc83f3c4" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39755 } }
I20260925 04:57:56.822038 21279 sys_catalog.cc:564] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:57:56.824386 21285 sys_catalog.cc:455] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e28737f93785414ab258f58efc83f3c4. Latest consensus state: current_term: 3 leader_uuid: "e28737f93785414ab258f58efc83f3c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28737f93785414ab258f58efc83f3c4" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39755 } } }
I20260925 04:57:56.824818 21285 sys_catalog.cc:458] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [sys.catalog]: This master's current role is: LEADER
I20260925 04:57:56.825484 21284 sys_catalog.cc:455] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e28737f93785414ab258f58efc83f3c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28737f93785414ab258f58efc83f3c4" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39755 } } }
I20260925 04:57:56.825824 21284 sys_catalog.cc:458] T 00000000000000000000000000000000 P e28737f93785414ab258f58efc83f3c4 [sys.catalog]: This master's current role is: LEADER
I20260925 04:57:56.835299 21289 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:57:56.845942 21289 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:57:56.848047 21289 catalog_manager.cc:1259] Loaded cluster ID: 9763c0e1d58a45b6b6c6faaeb1946feb
I20260925 04:57:56.848253 21289 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 04:57:56.850876 21289 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f383ed149ad google::LogMessage::Fail() at ??:0
@ 0x7f383ed1788a google::LogMessage::SendToLog() at ??:0
@ 0x7f383ed144de google::LogMessage::Flush() at ??:0
@ 0x7f383ed15a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f385474d512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f385474a976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
I20260925 04:57:56.883740 21302 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f38547d2610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f38547d2439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f3841960345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f384196391c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f3841963799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f384193eeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f3844c186db start_thread at ??:0
@ 0x7f383dbae61f clone at ??:0
W20260925 04:57:57.131910 21281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:57:57.132265 21281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:57:57.132431 21281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:57:57.140041 21281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:57:57.140192 21281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:57:57.140457 21281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.129
I20260925 04:57:57.151051 21281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:37825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.129:44167
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.29.129
--webserver_port=46169
--tserver_master_addrs=127.6.29.190:39755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.129
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:57:57.152500 21281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:57:57.153877 21281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:57:57.167385 21320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:57.167291 21321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:57.167363 21323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:57:57.168583 21281 server_base.cc:1034] running on GCE node
I20260925 04:57:57.169173 21281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:57:57.170091 21281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:57:57.171298 21281 hybrid_clock.cc:648] HybridClock initialized: now 1790312277171253 us; error 37 us; skew 500 ppm
I20260925 04:57:57.171614 21281 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:57:57 dist-test-slave-xzjr krb5kdc[20126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312277, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:57:57.174901 21281 init.cc:377] Logged in from keytab as kudu/127.6.29.129@KRBTEST.COM (short username kudu)
I20260925 04:57:57.176412 21281 webserver.cc:460] Webserver started at http://127.6.29.129:46169/ using document root <none> and password file <none>
I20260925 04:57:57.177151 21281 fs_manager.cc:359] Metadata directory not provided
I20260925 04:57:57.177279 21281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:57:57.182647 21281 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 04:57:57.185570 21330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:57.186936 21281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 04:57:57.187111 21281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/wal
uuid: "4a940a2e05694251b159ccbb2c4dda1c"
format_stamp: "Formatted at 2026-09-25 04:57:51 on dist-test-slave-xzjr"
server_key: "4afd8bcd8f07cdc64bd1749685dc7483"
server_key_iv: "69da25b82495cf436cc9170905d8fe96"
server_key_version: "encryptionkey@0"
I20260925 04:57:57.188138 21281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:57:57.203109 21281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:57:57.206974 21281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:57:57.207242 21281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:57:57.208259 21281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:57:57.210146 21337 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260925 04:57:57.215890 21281 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 04:57:57.216030 21281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260925 04:57:57.216159 21281 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 04:57:57.219280 21281 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 04:57:57.219379 21281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260925 04:57:57.219687 21337 tablet_bootstrap.cc:492] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c: Bootstrap starting.
I20260925 04:57:57.227811 21337 log.cc:824] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c: Log is configured to *not* fsync() on all Append() calls
I20260925 04:57:57.243221 21337 tablet_bootstrap.cc:492] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:57:57.243971 21337 tablet_bootstrap.cc:492] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c: Bootstrap complete.
I20260925 04:57:57.244457 21337 ts_tablet_manager.cc:1397] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c: Time spent bootstrapping tablet: real 0.025s user 0.013s sys 0.010s
I20260925 04:57:57.251753 21337 raft_consensus.cc:348] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff58da6112a4e8ba899f50028dd2b52" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 46361 } } peers { permanent_uuid: "4a940a2e05694251b159ccbb2c4dda1c" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 44167 } } peers { permanent_uuid: "e59fc77557484dd4a51d755847101947" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35163 } }
I20260925 04:57:57.253468 21337 raft_consensus.cc:729] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a940a2e05694251b159ccbb2c4dda1c, State: Initialized, Role: FOLLOWER
I20260925 04:57:57.254451 21337 consensus_queue.cc:260] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff58da6112a4e8ba899f50028dd2b52" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 46361 } } peers { permanent_uuid: "4a940a2e05694251b159ccbb2c4dda1c" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 44167 } } peers { permanent_uuid: "e59fc77557484dd4a51d755847101947" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35163 } }
I20260925 04:57:57.256498 21337 ts_tablet_manager.cc:1428] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c: Time spent starting tablet: real 0.012s user 0.010s sys 0.002s
I20260925 04:57:57.263487 21281 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.129:44167
I20260925 04:57:57.263561 21445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.129:44167 every 8 connection(s)
I20260925 04:57:57.264789 21281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
I20260925 04:57:57.269532 21448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:57.267532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.129:46235)
0925 04:57:57.268388 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:57.268647 (+ 259us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":382,"threads_started":1}
W20260925 04:57:57.270316 21446 heartbeater.cc:644] Failed to heartbeat to 127.6.29.190:39755 (0 consecutive failures): Network error: Failed to ping master at 127.6.29.190:39755: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
I20260925 04:57:57.270603 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 21281
I20260925 04:57:57.271356 21448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:57.270955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.129:35013)
0925 04:57:57.271125 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:57.271221 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 04:57:57.271332 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.130:35163
--local_ip_for_outbound_sockets=127.6.29.130
--tserver_master_addrs=127.6.29.190:39755
--webserver_port=39135
--webserver_interface=127.6.29.130
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:37825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:57:57.272375 21448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:57.271937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.129:56035)
0925 04:57:57.272107 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:57.272208 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 04:57:57.272637 21446 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 04:57:57.601229 21449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:57:57.601527 21449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:57:57.601722 21449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:57:57.609161 21449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:57:57.609297 21449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:57:57.609521 21449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.130
I20260925 04:57:57.619527 21449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:37825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.130:35163
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.29.130
--webserver_port=39135
--tserver_master_addrs=127.6.29.190:39755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.130
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:57:57.620985 21449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:57:57.622373 21449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:57:57.635921 21455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:57.635921 21457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:57.636054 21454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:57:57.641599 21449 server_base.cc:1034] running on GCE node
I20260925 04:57:57.642333 21449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:57:57.643461 21449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:57:57.644703 21449 hybrid_clock.cc:648] HybridClock initialized: now 1790312277644635 us; error 65 us; skew 500 ppm
I20260925 04:57:57.645109 21449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:57:57 dist-test-slave-xzjr krb5kdc[20126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312277, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:57:57.648550 21449 init.cc:377] Logged in from keytab as kudu/127.6.29.130@KRBTEST.COM (short username kudu)
I20260925 04:57:57.650298 21449 webserver.cc:460] Webserver started at http://127.6.29.130:39135/ using document root <none> and password file <none>
I20260925 04:57:57.651081 21449 fs_manager.cc:359] Metadata directory not provided
I20260925 04:57:57.651233 21449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:57:57.657157 21449 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 04:57:57.660136 21464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:57.661566 21449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 04:57:57.661762 21449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/wal
uuid: "e59fc77557484dd4a51d755847101947"
format_stamp: "Formatted at 2026-09-25 04:57:51 on dist-test-slave-xzjr"
server_key: "c38c87be4c7f1bb98772732cf809c71e"
server_key_iv: "c6ffd68bd8c9bf3d907b51b48104017a"
server_key_version: "encryptionkey@0"
I20260925 04:57:57.662813 21449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:57:57.689924 21449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:57:57.694712 21449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:57:57.695007 21449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:57:57.696043 21449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:57:57.697449 21449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:57:57.697573 21449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:57.697703 21449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:57:57.697788 21449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:57.731508 21449 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.130:35163
I20260925 04:57:57.731599 21576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.130:35163 every 8 connection(s)
I20260925 04:57:57.733083 21449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
I20260925 04:57:57.733986 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 21449
I20260925 04:57:57.734774 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.131:46361
--local_ip_for_outbound_sockets=127.6.29.131
--tserver_master_addrs=127.6.29.190:39755
--webserver_port=42127
--webserver_interface=127.6.29.131
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:37825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:57:57.738802 21579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:57.736410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.130:56023)
0925 04:57:57.737262 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:57.737547 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":300,"threads_started":1}
W20260925 04:57:57.739593 21577 heartbeater.cc:644] Failed to heartbeat to 127.6.29.190:39755 (0 consecutive failures): Network error: Failed to ping master at 127.6.29.190:39755: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
I20260925 04:57:57.740592 21579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:57.740149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.130:54765)
0925 04:57:57.740341 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:57.740439 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 04:57:57.741523 21579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:57.741110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.130:44087)
0925 04:57:57.741275 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:57.741373 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 04:57:57.741756 21577 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 04:57:58.066871 21580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:57:58.067173 21580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:57:58.067348 21580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:57:58.074698 21580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:57:58.074831 21580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:57:58.075017 21580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.131
I20260925 04:57:58.084810 21580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:37825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.131:46361
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.29.131
--webserver_port=42127
--tserver_master_addrs=127.6.29.190:39755
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.131
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:57:58.086436 21580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:57:58.087936 21580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:57:58.101485 21586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:58.101485 21585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:57:58.101514 21588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:57:58.102082 21580 server_base.cc:1034] running on GCE node
I20260925 04:57:58.102773 21580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:57:58.104041 21580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:57:58.105262 21580 hybrid_clock.cc:648] HybridClock initialized: now 1790312278105219 us; error 41 us; skew 500 ppm
I20260925 04:57:58.105577 21580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:57:58 dist-test-slave-xzjr krb5kdc[20126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312278, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:57:58.108806 21580 init.cc:377] Logged in from keytab as kudu/127.6.29.131@KRBTEST.COM (short username kudu)
I20260925 04:57:58.110285 21580 webserver.cc:460] Webserver started at http://127.6.29.131:42127/ using document root <none> and password file <none>
I20260925 04:57:58.110996 21580 fs_manager.cc:359] Metadata directory not provided
I20260925 04:57:58.111142 21580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:57:58.116699 21580 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 04:57:58.119431 21595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:58.120744 21580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 04:57:58.120914 21580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/wal
uuid: "dff58da6112a4e8ba899f50028dd2b52"
format_stamp: "Formatted at 2026-09-25 04:57:52 on dist-test-slave-xzjr"
server_key: "b8ea3f81090e2492de138248abf6e460"
server_key_iv: "756f0ea0011f3845c45d63fb11b289ab"
server_key_version: "encryptionkey@0"
I20260925 04:57:58.121877 21580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:57:58.137013 21580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:57:58.141073 21580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:57:58.141345 21580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:57:58.142308 21580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:57:58.143612 21580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:57:58.143734 21580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:58.143879 21580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:57:58.143965 21580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:57:58.177526 21580 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.131:46361
I20260925 04:57:58.177632 21707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.131:46361 every 8 connection(s)
I20260925 04:57:58.179045 21580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
I20260925 04:57:58.184235 21710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:58.182333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.131:39231)
0925 04:57:58.183078 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:58.183281 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":294,"threads_started":1}
W20260925 04:57:58.184959 21708 heartbeater.cc:644] Failed to heartbeat to 127.6.29.190:39755 (0 consecutive failures): Network error: Failed to ping master at 127.6.29.190:39755: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
I20260925 04:57:58.185738 21710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:58.185451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.131:35183)
0925 04:57:58.185577 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:58.185642 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 04:57:58.186504 21710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:58.186225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.131:35383)
0925 04:57:58.186325 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:58.186389 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 04:57:58.186751 21708 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 04:57:58.188882 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 21580
I20260925 04:57:58.273960 21712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:58.273196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.129:42847)
0925 04:57:58.273642 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:58.273728 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":212,"threads_started":1}
I20260925 04:57:58.473981 21713 raft_consensus.cc:482] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:57:58.474319 21713 raft_consensus.cc:504] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff58da6112a4e8ba899f50028dd2b52" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 46361 } } peers { permanent_uuid: "4a940a2e05694251b159ccbb2c4dda1c" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 44167 } } peers { permanent_uuid: "e59fc77557484dd4a51d755847101947" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35163 } }
I20260925 04:57:58.476045 21713 leader_election.cc:288] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff58da6112a4e8ba899f50028dd2b52 (127.6.29.131:46361), e59fc77557484dd4a51d755847101947 (127.6.29.130:35163)
Sep 25 04:57:58 dist-test-slave-xzjr krb5kdc[20126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312277, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for kudu/127.6.29.130@KRBTEST.COM
Sep 25 04:57:58 dist-test-slave-xzjr krb5kdc[20126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312277, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for kudu/127.6.29.131@KRBTEST.COM
I20260925 04:57:58.499270 21712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:58.476254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.131:46361 (local address 127.6.29.129:49159)
0925 04:57:58.476457 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:58.476494 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 04:57:58.477101 (+ 607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:57:58.481093 (+ 3992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:57:58.481100 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:57:58.481757 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:57:58.482521 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:58.482543 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:58.484148 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:58.484151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:58.485591 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:58.485603 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:58.486568 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:58.486575 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:58.486888 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:58.487670 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:57:58.487695 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:57:58.490584 (+ 2889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:57:58.495293 (+ 4709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:58.495301 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:58.495313 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:58.495824 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:58.496299 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:58.496306 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:58.496312 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:58.496576 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:58.497569 (+ 993us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:57:58.497575 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:57:58.497881 (+ 306us) client_negotiation.cc:769] Sending connection context
0925 04:57:58.498233 (+ 352us) client_negotiation.cc:240] Negotiation successful
0925 04:57:58.498808 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":192,"spinlock_wait_cycles":4480}
I20260925 04:57:58.499382 21715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:58.476666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:49159 (local address 127.6.29.131:46361)
0925 04:57:58.478469 (+ 1803us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:58.478487 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:58.478533 (+ 46us) server_negotiation.cc:407] Connection header received
0925 04:57:58.480070 (+ 1537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:58.480079 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:58.480616 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:58.481815 (+ 1199us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:58.482716 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:58.483964 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:58.485727 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:58.486352 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:58.486543 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:58.490878 (+ 4335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:58.490888 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:58.490902 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:58.490978 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:58.495134 (+ 4156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:58.495978 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:58.495985 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:58.495992 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:58.496170 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:58.496688 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:58.496694 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:58.496699 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:58.497431 (+ 732us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:58.497617 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:58.498498 (+ 881us) server_negotiation.cc:299] Negotiation successful
0925 04:57:58.498934 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1626,"thread_start_us":1432,"threads_started":1}
I20260925 04:57:58.499392 21716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:58.476978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:41097 (local address 127.6.29.130:35163)
0925 04:57:58.477933 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0925 04:57:58.477952 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 04:57:58.478402 (+ 450us) server_negotiation.cc:407] Connection header received
0925 04:57:58.480212 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:57:58.480222 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:57:58.480653 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:57:58.481022 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:57:58.482724 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:58.483970 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:58.485740 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:57:58.486375 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:57:58.486561 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:58.490754 (+ 4193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:57:58.490768 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:57:58.490791 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:57:58.490846 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:57:58.495134 (+ 4288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:58.496612 (+ 1478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:58.496618 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:58.496623 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:58.496872 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:57:58.497224 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:57:58.497228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:57:58.497231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:57:58.497892 (+ 661us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:57:58.498024 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 04:57:58.498562 (+ 538us) server_negotiation.cc:299] Negotiation successful
0925 04:57:58.498934 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":766,"thread_start_us":647,"threads_started":1}
I20260925 04:57:58.499331 21714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:58.476679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:35163 (local address 127.6.29.129:41097)
0925 04:57:58.478272 (+ 1593us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:58.478293 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 04:57:58.478435 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:57:58.481095 (+ 2660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:57:58.481102 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:57:58.481801 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:57:58.482519 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:58.482543 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:58.484142 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:58.484147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:58.485596 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:57:58.485608 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:57:58.486585 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:57:58.486592 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:57:58.486905 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:57:58.487748 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:57:58.487822 (+ 74us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:57:58.490375 (+ 2553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:57:58.495996 (+ 5621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:58.496003 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:58.496006 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:58.496479 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:58.496993 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:57:58.496997 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:57:58.496999 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:57:58.497122 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:57:58.498024 (+ 902us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:57:58.498030 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:57:58.498174 (+ 144us) client_negotiation.cc:769] Sending connection context
0925 04:57:58.498334 (+ 160us) client_negotiation.cc:240] Negotiation successful
0925 04:57:58.498934 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1444,"mutex_wait_us":362,"spinlock_wait_cycles":35200,"thread_start_us":1265,"threads_started":1}
W20260925 04:57:58.502882 21333 leader_election.cc:341] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e59fc77557484dd4a51d755847101947 (127.6.29.130:35163): Not found: Tablet not found: 23fec7a7429b4884ac342d0319e6c0ca
W20260925 04:57:58.503240 21333 leader_election.cc:341] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dff58da6112a4e8ba899f50028dd2b52 (127.6.29.131:46361): Not found: Tablet not found: 23fec7a7429b4884ac342d0319e6c0ca
I20260925 04:57:58.503418 21333 leader_election.cc:302] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a940a2e05694251b159ccbb2c4dda1c; no voters: dff58da6112a4e8ba899f50028dd2b52, e59fc77557484dd4a51d755847101947
I20260925 04:57:58.503855 21713 raft_consensus.cc:2726] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 04:57:58.743134 21717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:58.742281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.130:34523)
0925 04:57:58.742747 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:58.742895 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":226,"threads_started":1}
I20260925 04:57:59.188153 21718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:59.187280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.131:57681)
0925 04:57:59.187688 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:59.187958 (+ 270us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":206,"threads_started":1}
I20260925 04:57:59.275751 21719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:59.274947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.129:57133)
0925 04:57:59.275525 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:59.275601 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":311,"threads_started":1}
I20260925 04:57:59.744967 21720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:57:59.744037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.130:58939)
0925 04:57:59.744627 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0925 04:57:59.744740 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":327,"threads_started":1}
I20260925 04:58:00.189716 21721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:00.189030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.131:35413)
0925 04:58:00.189455 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:00.189539 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":215,"threads_started":1}
I20260925 04:58:00.277352 21722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:00.276663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.129:58459)
0925 04:58:00.277097 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:00.277179 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":220,"threads_started":1}
I20260925 04:58:00.722635 21723 raft_consensus.cc:482] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:58:00.722862 21723 raft_consensus.cc:504] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff58da6112a4e8ba899f50028dd2b52" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 46361 } } peers { permanent_uuid: "4a940a2e05694251b159ccbb2c4dda1c" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 44167 } } peers { permanent_uuid: "e59fc77557484dd4a51d755847101947" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35163 } }
I20260925 04:58:00.723537 21723 leader_election.cc:288] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff58da6112a4e8ba899f50028dd2b52 (127.6.29.131:46361), e59fc77557484dd4a51d755847101947 (127.6.29.130:35163)
W20260925 04:58:00.724644 21333 leader_election.cc:341] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dff58da6112a4e8ba899f50028dd2b52 (127.6.29.131:46361): Not found: Tablet not found: 23fec7a7429b4884ac342d0319e6c0ca
W20260925 04:58:00.724872 21333 leader_election.cc:341] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e59fc77557484dd4a51d755847101947 (127.6.29.130:35163): Not found: Tablet not found: 23fec7a7429b4884ac342d0319e6c0ca
I20260925 04:58:00.724968 21333 leader_election.cc:302] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a940a2e05694251b159ccbb2c4dda1c; no voters: dff58da6112a4e8ba899f50028dd2b52, e59fc77557484dd4a51d755847101947
I20260925 04:58:00.725246 21723 raft_consensus.cc:2726] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 04:58:00.746681 21724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:00.745953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.130:41311)
0925 04:58:00.746424 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:00.746514 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":238,"threads_started":1}
I20260925 04:58:01.191417 21725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:01.190613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.131:47973)
0925 04:58:01.191142 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:01.191241 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":268,"threads_started":1}
I20260925 04:58:01.279047 21726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:01.278356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.129:52745)
0925 04:58:01.278792 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:01.278874 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":225,"threads_started":1}
I20260925 04:58:01.748463 21727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:01.747647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.130:57593)
0925 04:58:01.748181 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:01.748282 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":300,"threads_started":1}
I20260925 04:58:02.193073 21728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:02.192274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.131:47045)
0925 04:58:02.192807 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:02.192890 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":309,"threads_started":1}
I20260925 04:58:02.280999 21729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:02.280206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.129:49029)
0925 04:58:02.280724 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:02.280812 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":251,"threads_started":1}
I20260925 04:58:02.750087 21730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:02.749397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.130:38507)
0925 04:58:02.749835 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:02.749913 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":233,"threads_started":1}
I20260925 04:58:03.068624 21731 raft_consensus.cc:482] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:58:03.068900 21731 raft_consensus.cc:504] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff58da6112a4e8ba899f50028dd2b52" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 46361 } } peers { permanent_uuid: "4a940a2e05694251b159ccbb2c4dda1c" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 44167 } } peers { permanent_uuid: "e59fc77557484dd4a51d755847101947" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35163 } }
I20260925 04:58:03.069610 21731 leader_election.cc:288] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff58da6112a4e8ba899f50028dd2b52 (127.6.29.131:46361), e59fc77557484dd4a51d755847101947 (127.6.29.130:35163)
W20260925 04:58:03.070755 21333 leader_election.cc:341] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dff58da6112a4e8ba899f50028dd2b52 (127.6.29.131:46361): Not found: Tablet not found: 23fec7a7429b4884ac342d0319e6c0ca
W20260925 04:58:03.071000 21333 leader_election.cc:341] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e59fc77557484dd4a51d755847101947 (127.6.29.130:35163): Not found: Tablet not found: 23fec7a7429b4884ac342d0319e6c0ca
I20260925 04:58:03.071122 21333 leader_election.cc:302] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a940a2e05694251b159ccbb2c4dda1c; no voters: dff58da6112a4e8ba899f50028dd2b52, e59fc77557484dd4a51d755847101947
I20260925 04:58:03.071487 21731 raft_consensus.cc:2726] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 04:58:03.194859 21732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:03.194055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.131:40917)
0925 04:58:03.194614 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:03.194695 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":309,"threads_started":1}
I20260925 04:58:03.282685 21733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:03.281905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.129:36931)
0925 04:58:03.282379 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:03.282496 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":230,"threads_started":1}
I20260925 04:58:03.751948 21734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:03.751054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.130:40435)
0925 04:58:03.751631 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:03.751709 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":346,"threads_started":1}
I20260925 04:58:04.196560 21735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:04.195732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.131:49197)
0925 04:58:04.196304 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:04.196383 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":293,"threads_started":1}
I20260925 04:58:04.284554 21736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:04.283622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.129:48173)
0925 04:58:04.284181 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:04.284306 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":301,"threads_started":1}
I20260925 04:58:04.753983 21737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:04.753015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.130:45721)
0925 04:58:04.753695 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:04.753794 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":313,"threads_started":1}
I20260925 04:58:05.198271 21738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:05.197450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.131:49059)
0925 04:58:05.197958 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:05.198040 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":232,"threads_started":1}
I20260925 04:58:05.286257 21739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:05.285520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.129:59595)
0925 04:58:05.286002 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:05.286086 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":261,"threads_started":1}
I20260925 04:58:05.755697 21740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:05.755022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.130:45669)
0925 04:58:05.755434 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:05.755512 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":211,"threads_started":1}
I20260925 04:58:05.831862 21741 raft_consensus.cc:482] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:58:05.832082 21741 raft_consensus.cc:504] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff58da6112a4e8ba899f50028dd2b52" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 46361 } } peers { permanent_uuid: "4a940a2e05694251b159ccbb2c4dda1c" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 44167 } } peers { permanent_uuid: "e59fc77557484dd4a51d755847101947" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35163 } }
I20260925 04:58:05.832820 21741 leader_election.cc:288] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff58da6112a4e8ba899f50028dd2b52 (127.6.29.131:46361), e59fc77557484dd4a51d755847101947 (127.6.29.130:35163)
W20260925 04:58:05.833732 21333 leader_election.cc:341] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dff58da6112a4e8ba899f50028dd2b52 (127.6.29.131:46361): Not found: Tablet not found: 23fec7a7429b4884ac342d0319e6c0ca
W20260925 04:58:05.833927 21333 leader_election.cc:341] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e59fc77557484dd4a51d755847101947 (127.6.29.130:35163): Not found: Tablet not found: 23fec7a7429b4884ac342d0319e6c0ca
I20260925 04:58:05.833999 21333 leader_election.cc:302] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a940a2e05694251b159ccbb2c4dda1c; no voters: dff58da6112a4e8ba899f50028dd2b52, e59fc77557484dd4a51d755847101947
I20260925 04:58:05.834225 21741 raft_consensus.cc:2726] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 04:58:06.200158 21742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:06.199332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.131:38737)
0925 04:58:06.199855 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:06.199965 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":239,"threads_started":1}
I20260925 04:58:06.288254 21743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:06.287238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.129:33329)
0925 04:58:06.287816 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:06.287975 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":250,"threads_started":1}
I20260925 04:58:06.757725 21744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:06.756819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.130:37749)
0925 04:58:06.757468 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:06.757567 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":299,"threads_started":1}
I20260925 04:58:07.202133 21745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:07.201337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.131:52481)
0925 04:58:07.201852 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:07.201947 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":252,"threads_started":1}
I20260925 04:58:07.290365 21746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:07.289583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.129:33771)
0925 04:58:07.290089 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:07.290191 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":263,"threads_started":1}
I20260925 04:58:07.759567 21747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:07.758812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.130:34201)
0925 04:58:07.759294 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:07.759383 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":261,"threads_started":1}
I20260925 04:58:08.204092 21748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:08.203172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.131:58569)
0925 04:58:08.203818 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:08.203913 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":392,"threads_started":1}
I20260925 04:58:08.292685 21749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:08.291571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.129:43383)
0925 04:58:08.292213 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:08.292322 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":306,"threads_started":1}
I20260925 04:58:08.761500 21750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:08.760698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.130:58031)
0925 04:58:08.761198 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:08.761332 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":249,"threads_started":1}
I20260925 04:58:09.205826 21751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:09.205062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.131:58633)
0925 04:58:09.205567 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:09.205658 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":243,"threads_started":1}
I20260925 04:58:09.294484 21752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:09.293764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.129:50101)
0925 04:58:09.294232 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:09.294316 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":237,"threads_started":1}
I20260925 04:58:09.763696 21753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:09.762808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.130:44447)
0925 04:58:09.763372 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:09.763512 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":297,"threads_started":1}
I20260925 04:58:10.207863 21754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:10.207031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.131:60949)
0925 04:58:10.207530 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:10.207619 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":268,"threads_started":1}
I20260925 04:58:10.296329 21755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:10.295425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.129:41567)
0925 04:58:10.296046 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:10.296139 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":229,"threads_started":1}
I20260925 04:58:10.436342 21756 raft_consensus.cc:482] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:58:10.436607 21756 raft_consensus.cc:504] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff58da6112a4e8ba899f50028dd2b52" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 46361 } } peers { permanent_uuid: "4a940a2e05694251b159ccbb2c4dda1c" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 44167 } } peers { permanent_uuid: "e59fc77557484dd4a51d755847101947" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35163 } }
I20260925 04:58:10.437397 21756 leader_election.cc:288] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff58da6112a4e8ba899f50028dd2b52 (127.6.29.131:46361), e59fc77557484dd4a51d755847101947 (127.6.29.130:35163)
W20260925 04:58:10.438350 21333 leader_election.cc:341] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dff58da6112a4e8ba899f50028dd2b52 (127.6.29.131:46361): Not found: Tablet not found: 23fec7a7429b4884ac342d0319e6c0ca
W20260925 04:58:10.438596 21333 leader_election.cc:341] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e59fc77557484dd4a51d755847101947 (127.6.29.130:35163): Not found: Tablet not found: 23fec7a7429b4884ac342d0319e6c0ca
I20260925 04:58:10.438722 21333 leader_election.cc:302] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a940a2e05694251b159ccbb2c4dda1c; no voters: dff58da6112a4e8ba899f50028dd2b52, e59fc77557484dd4a51d755847101947
I20260925 04:58:10.439021 21756 raft_consensus.cc:2726] T 23fec7a7429b4884ac342d0319e6c0ca P 4a940a2e05694251b159ccbb2c4dda1c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 04:58:10.765599 21757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:10.764745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.130:55413)
0925 04:58:10.765322 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:10.765426 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":260,"threads_started":1}
I20260925 04:58:11.209712 21758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:11.208955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.131:43319)
0925 04:58:11.209450 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:11.209548 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":237,"threads_started":1}
I20260925 04:58:11.298187 21759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:11.297389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.129:40441)
0925 04:58:11.297894 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:11.297995 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":247,"threads_started":1}
I20260925 04:58:11.767964 21760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:11.766770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.130:43359)
0925 04:58:11.767458 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:11.767552 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":312,"threads_started":1}
I20260925 04:58:12.211637 21761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:12.210817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.131:39861)
0925 04:58:12.211347 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:12.211438 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":275,"threads_started":1}
I20260925 04:58:12.300536 21762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:12.299587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.129:46975)
0925 04:58:12.300233 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:12.300333 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":396,"threads_started":1}
I20260925 04:58:12.769924 21763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:12.769191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.130:59827)
0925 04:58:12.769631 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:12.769740 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":232,"threads_started":1}
I20260925 04:58:13.193490 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 21281
I20260925 04:58:13.210112 21764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:13.194892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51144 (local address 127.6.29.129:44167)
0925 04:58:13.195424 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0925 04:58:13.195431 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:58:13.195457 (+ 26us) server_negotiation.cc:407] Connection header received
0925 04:58:13.195520 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:58:13.195530 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:58:13.195829 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:58:13.195978 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:58:13.197663 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:13.198733 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:13.200301 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:13.200925 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:13.201045 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:13.203952 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:58:13.203968 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:58:13.203984 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:58:13.204030 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:58:13.207078 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:58:13.207978 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:58:13.207987 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:58:13.207990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:58:13.208137 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:58:13.208586 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:58:13.208590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:58:13.208592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:58:13.208975 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:58:13.209204 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 04:58:13.209598 (+ 394us) server_negotiation.cc:299] Negotiation successful
0925 04:58:13.209838 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":230,"threads_started":1}
I20260925 04:58:13.211498 21441 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:58:13.213927 21766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:13.212946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:39755 (local address 127.6.29.131:59535)
0925 04:58:13.213511 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:13.213678 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:39755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":259,"threads_started":1}
I20260925 04:58:13.304298 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 21281
I20260925 04:58:13.321909 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 21449
I20260925 04:58:13.336232 21768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:13.322728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37042 (local address 127.6.29.130:35163)
0925 04:58:13.323268 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0925 04:58:13.323273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:58:13.323289 (+ 16us) server_negotiation.cc:407] Connection header received
0925 04:58:13.323348 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:58:13.323352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:58:13.323419 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:58:13.323513 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:58:13.325027 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:13.325903 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:13.327191 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:13.327651 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:13.327751 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:13.330651 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:58:13.330656 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:58:13.330660 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:58:13.330700 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:58:13.333574 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:58:13.334308 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:58:13.334312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:58:13.334314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:58:13.334497 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:58:13.334831 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:58:13.334834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:58:13.334836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:58:13.335116 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:58:13.335325 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 04:58:13.335671 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 04:58:13.335944 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":301,"threads_started":1}
I20260925 04:58:13.337008 21572 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:58:13.424096 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 21449
I20260925 04:58:13.443133 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 21580
I20260925 04:58:13.459077 21770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:13.444072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36832 (local address 127.6.29.131:46361)
0925 04:58:13.444649 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0925 04:58:13.444654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:58:13.444672 (+ 18us) server_negotiation.cc:407] Connection header received
0925 04:58:13.444728 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:58:13.444740 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:58:13.444813 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:58:13.444910 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:58:13.446431 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:13.447286 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:13.448972 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:13.449454 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:13.449575 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:13.452629 (+ 3054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:58:13.452635 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:58:13.452639 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:58:13.452679 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:58:13.455672 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:58:13.457043 (+ 1371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:58:13.457048 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:58:13.457051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:58:13.457218 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:58:13.457664 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:58:13.457669 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:58:13.457671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:58:13.458013 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:58:13.458252 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0925 04:58:13.458578 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 04:58:13.458766 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":316,"threads_started":1}
I20260925 04:58:13.460001 21703 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:58:13.547652 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 21580
2026-09-25T04:58:13Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24521 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:58:13 dist-test-slave-xzjr krb5kdc[21774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:58:13 dist-test-slave-xzjr krb5kdc[21774](info): set up 2 sockets
Sep 25 04:58:13 dist-test-slave-xzjr krb5kdc[21774](info): commencing operation
krb5kdc: starting...
W20260925 04:58:14.673022 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 04:58:14 dist-test-slave-xzjr krb5kdc[21774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:58:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:58:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.29.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
I20260925 04:58:14.940501 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:45703
--webserver_interface=127.6.29.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:34331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:45703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:58:15.284855 21787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:58:15.285171 21787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:58:15.285356 21787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:58:15.292899 21787 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 04:58:15.293015 21787 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 04:58:15.293053 21787 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 04:58:15.293156 21787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:58:15.293267 21787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:58:15.293381 21787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:58:15.293463 21787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:58:15.293547 21787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:58:15.303916 21787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:34331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:45703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:45703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:58:15.305343 21787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:58:15.306741 21787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:58:15.319307 21792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:58:15.319242 21795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:58:15.319196 21793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:58:15.319653 21787 server_base.cc:1034] running on GCE node
I20260925 04:58:15.320860 21787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:58:15.322350 21787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:58:15.323575 21787 hybrid_clock.cc:648] HybridClock initialized: now 1790312295323500 us; error 52 us; skew 500 ppm
I20260925 04:58:15.324014 21787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:58:15 dist-test-slave-xzjr krb5kdc[21774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312295, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:58:15.332801 21787 init.cc:377] Logged in from keytab as kudu/127.6.29.190@KRBTEST.COM (short username kudu)
I20260925 04:58:15.334434 21787 webserver.cc:460] Webserver started at http://127.6.29.190:46627/ using document root <none> and password file <none>
I20260925 04:58:15.335141 21787 fs_manager.cc:359] Metadata directory not provided
I20260925 04:58:15.335233 21787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:58:15.335518 21787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:58:15.338743 21787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "c42d92ab26a24f459a6f7dd1b623038b"
format_stamp: "Formatted at 2026-09-25 04:58:15 on dist-test-slave-xzjr"
server_key: "c9b22eb5243941d99b53fa7fd6ed3e91"
server_key_iv: "12720e2e4b5487fbb25b272d43b033e0"
server_key_version: "encryptionkey@0"
I20260925 04:58:15.339527 21787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "c42d92ab26a24f459a6f7dd1b623038b"
format_stamp: "Formatted at 2026-09-25 04:58:15 on dist-test-slave-xzjr"
server_key: "c9b22eb5243941d99b53fa7fd6ed3e91"
server_key_iv: "12720e2e4b5487fbb25b272d43b033e0"
server_key_version: "encryptionkey@0"
I20260925 04:58:15.345089 21787 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 04:58:15.348862 21802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:58:15.350382 21787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:58:15.350526 21787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "c42d92ab26a24f459a6f7dd1b623038b"
format_stamp: "Formatted at 2026-09-25 04:58:15 on dist-test-slave-xzjr"
server_key: "c9b22eb5243941d99b53fa7fd6ed3e91"
server_key_iv: "12720e2e4b5487fbb25b272d43b033e0"
server_key_version: "encryptionkey@0"
I20260925 04:58:15.350713 21787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:58:15.390025 21787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:58:15.394850 21787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:58:15.395118 21787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:58:15.416078 21787 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:45703
I20260925 04:58:15.416105 21853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:45703 every 8 connection(s)
I20260925 04:58:15.417837 21787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:58:15.422072 21854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:58:15.426875 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 21787
I20260925 04:58:15.427107 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 04:58:15.427852 6262 external_mini_cluster.cc:1428] Setting key e398049f0e136bf3b179d055fcc714bb
I20260925 04:58:15.433779 21854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b: Bootstrap starting.
I20260925 04:58:15.437419 21854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b: Neither blocks nor log segments found. Creating new log.
I20260925 04:58:15.438515 21854 log.cc:824] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b: Log is configured to *not* fsync() on all Append() calls
I20260925 04:58:15.442237 21854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b: No bootstrap required, opened a new log
Sep 25 04:58:15 dist-test-slave-xzjr krb5kdc[21774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:58:15.453310 21854 raft_consensus.cc:348] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42d92ab26a24f459a6f7dd1b623038b" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 45703 } }
I20260925 04:58:15.453841 21854 raft_consensus.cc:374] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:58:15.453960 21854 raft_consensus.cc:729] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c42d92ab26a24f459a6f7dd1b623038b, State: Initialized, Role: FOLLOWER
I20260925 04:58:15.454201 21857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:15.430019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44030 (local address 127.6.29.190:45703)
0925 04:58:15.430880 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0925 04:58:15.430896 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:58:15.430944 (+ 48us) server_negotiation.cc:407] Connection header received
0925 04:58:15.432824 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:58:15.432849 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:58:15.433265 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:58:15.433684 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:58:15.435222 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:15.436686 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:15.438157 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:15.438938 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:15.439177 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:15.444629 (+ 5452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:58:15.444641 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:58:15.444662 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:58:15.444720 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:58:15.449036 (+ 4316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:58:15.449828 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:58:15.449837 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:58:15.449849 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:58:15.450105 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:58:15.450516 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:58:15.450521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:58:15.450526 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:58:15.451368 (+ 842us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:58:15.451697 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0925 04:58:15.452180 (+ 483us) server_negotiation.cc:299] Negotiation successful
0925 04:58:15.452665 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":385,"threads_started":1}
I20260925 04:58:15.454712 21854 consensus_queue.cc:260] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42d92ab26a24f459a6f7dd1b623038b" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 45703 } }
I20260925 04:58:15.454989 21854 raft_consensus.cc:388] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:58:15.455163 21854 raft_consensus.cc:482] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:58:15.455348 21854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:58:15.457860 21854 raft_consensus.cc:504] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42d92ab26a24f459a6f7dd1b623038b" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 45703 } }
I20260925 04:58:15.458380 21854 leader_election.cc:302] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c42d92ab26a24f459a6f7dd1b623038b; no voters:
I20260925 04:58:15.458951 21854 leader_election.cc:288] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:58:15.459112 21859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:58:15.459865 21859 raft_consensus.cc:686] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [term 1 LEADER]: Becoming Leader. State: Replica: c42d92ab26a24f459a6f7dd1b623038b, State: Running, Role: LEADER
I20260925 04:58:15.460454 21859 consensus_queue.cc:237] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42d92ab26a24f459a6f7dd1b623038b" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 45703 } }
I20260925 04:58:15.461059 21854 sys_catalog.cc:564] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:58:15.463500 21861 sys_catalog.cc:455] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c42d92ab26a24f459a6f7dd1b623038b. Latest consensus state: current_term: 1 leader_uuid: "c42d92ab26a24f459a6f7dd1b623038b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42d92ab26a24f459a6f7dd1b623038b" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 45703 } } }
I20260925 04:58:15.463881 21861 sys_catalog.cc:458] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [sys.catalog]: This master's current role is: LEADER
I20260925 04:58:15.464908 21860 sys_catalog.cc:455] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c42d92ab26a24f459a6f7dd1b623038b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42d92ab26a24f459a6f7dd1b623038b" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 45703 } } }
I20260925 04:58:15.465510 21860 sys_catalog.cc:458] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [sys.catalog]: This master's current role is: LEADER
I20260925 04:58:15.466140 21865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:58:15.472465 21865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:58:15.482762 21865 catalog_manager.cc:1347] Generated new cluster ID: 23f56071d0524962a48d3f716d58e81a
I20260925 04:58:15.482920 21865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:58:15.506280 21865 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:58:15.507752 21865 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:58:15.516182 21865 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b: Generated new TSK 0
I20260925 04:58:15.517438 21865 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.29.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab.
Entry for principal kudu/127.6.29.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab.
I20260925 04:58:15.592953 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.129:0
--local_ip_for_outbound_sockets=127.6.29.129
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:45703
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:34331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:58:15.947511 21880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:58:15.947908 21880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:58:15.948086 21880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:58:15.955928 21880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:58:15.956089 21880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:58:15.956270 21880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.129
I20260925 04:58:15.966804 21880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:34331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:45703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.129
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:58:15.968469 21880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:58:15.969801 21880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:58:15.983548 21885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:58:15.983520 21888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:58:15.983475 21886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:58:15.984664 21880 server_base.cc:1034] running on GCE node
I20260925 04:58:15.985294 21880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:58:15.986292 21880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:58:15.987530 21880 hybrid_clock.cc:648] HybridClock initialized: now 1790312295987476 us; error 48 us; skew 500 ppm
I20260925 04:58:15.987919 21880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:58:15 dist-test-slave-xzjr krb5kdc[21774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312295, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:58:15.991376 21880 init.cc:377] Logged in from keytab as kudu/127.6.29.129@KRBTEST.COM (short username kudu)
I20260925 04:58:15.993000 21880 webserver.cc:460] Webserver started at http://127.6.29.129:40339/ using document root <none> and password file <none>
I20260925 04:58:15.993687 21880 fs_manager.cc:359] Metadata directory not provided
I20260925 04:58:15.993832 21880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:58:15.994143 21880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:58:15.997109 21880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/data/instance:
uuid: "7071476802644c14854c4de13332b96f"
format_stamp: "Formatted at 2026-09-25 04:58:15 on dist-test-slave-xzjr"
server_key: "6a769dbf6ab00d67342d9c3faac2768b"
server_key_iv: "0797aca69167b9a4f10bff35978bdd08"
server_key_version: "encryptionkey@0"
I20260925 04:58:15.997869 21880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance:
uuid: "7071476802644c14854c4de13332b96f"
format_stamp: "Formatted at 2026-09-25 04:58:15 on dist-test-slave-xzjr"
server_key: "6a769dbf6ab00d67342d9c3faac2768b"
server_key_iv: "0797aca69167b9a4f10bff35978bdd08"
server_key_version: "encryptionkey@0"
I20260925 04:58:16.003139 21880 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 04:58:16.006800 21895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:58:16.008441 21880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:58:16.008635 21880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/wal
uuid: "7071476802644c14854c4de13332b96f"
format_stamp: "Formatted at 2026-09-25 04:58:15 on dist-test-slave-xzjr"
server_key: "6a769dbf6ab00d67342d9c3faac2768b"
server_key_iv: "0797aca69167b9a4f10bff35978bdd08"
server_key_version: "encryptionkey@0"
I20260925 04:58:16.008853 21880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:58:16.040925 21880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:58:16.045696 21880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:58:16.046032 21880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:58:16.047065 21880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:58:16.048493 21880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:58:16.048599 21880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:58:16.048758 21880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:58:16.048851 21880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:58:16.084389 21880 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.129:46341
I20260925 04:58:16.084483 22007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.129:46341 every 8 connection(s)
I20260925 04:58:16.085780 21880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
I20260925 04:58:16.091255 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 21880
I20260925 04:58:16.091498 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance
I20260925 04:58:16.092242 6262 external_mini_cluster.cc:1428] Setting key 405cb795409a274d1e07b61580e85ca1
Sep 25 04:58:16 dist-test-slave-xzjr krb5kdc[21774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312295, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:58:16.108069 22010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:16.088794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:48593 (local address 127.6.29.190:45703)
0925 04:58:16.089255 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0925 04:58:16.089261 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:58:16.090431 (+ 1170us) server_negotiation.cc:407] Connection header received
0925 04:58:16.092616 (+ 2185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:58:16.092623 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:58:16.092721 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:58:16.092843 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:58:16.094751 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:16.095478 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:16.097174 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:16.097579 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:16.097713 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:16.101318 (+ 3605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:58:16.101324 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:58:16.101328 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:58:16.101362 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:58:16.104233 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:58:16.105319 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:58:16.105324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:58:16.105326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:58:16.105491 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:58:16.106062 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:58:16.106066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:58:16.106069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:58:16.106411 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:58:16.106543 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 04:58:16.107483 (+ 940us) server_negotiation.cc:299] Negotiation successful
0925 04:58:16.107673 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":227,"threads_started":1}
I20260925 04:58:16.109583 22011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:16.089448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.129:48593)
0925 04:58:16.090219 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:16.090259 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 04:58:16.092274 (+ 2015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:58:16.093121 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:58:16.093130 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:58:16.093932 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:58:16.094582 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:58:16.094601 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:58:16.095654 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:58:16.095658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:58:16.097026 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:58:16.097039 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:58:16.097835 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:58:16.097842 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:58:16.098088 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:16.098815 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:58:16.098839 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:58:16.101087 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:58:16.104396 (+ 3309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:58:16.104408 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:58:16.104429 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:58:16.105150 (+ 721us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:58:16.105611 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:58:16.105619 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:58:16.105626 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:58:16.105898 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:58:16.106560 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:58:16.106571 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:58:16.107070 (+ 499us) client_negotiation.cc:769] Sending connection context
0925 04:58:16.107381 (+ 311us) client_negotiation.cc:240] Negotiation successful
0925 04:58:16.108043 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":320,"threads_started":1}
I20260925 04:58:16.111351 22008 heartbeater.cc:347] Connected to a master server at 127.6.29.190:45703
I20260925 04:58:16.111740 22008 heartbeater.cc:464] Registering TS with master...
I20260925 04:58:16.112746 22008 heartbeater.cc:511] Master 127.6.29.190:45703 requested a full tablet report, sending...
I20260925 04:58:16.115332 21819 ts_manager.cc:194] Registered new tserver with Master: 7071476802644c14854c4de13332b96f (127.6.29.129:46341)
WARNING: no policy specified for kudu/127.6.29.130@KRBTEST.COM; defaulting to no policy
I20260925 04:58:16.117023 21819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.129@KRBTEST.COM'} at 127.6.29.129:48593
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab.
Entry for principal kudu/127.6.29.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab.
I20260925 04:58:16.151933 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.130:0
--local_ip_for_outbound_sockets=127.6.29.130
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:45703
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:34331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:58:16.512281 22014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:58:16.512713 22014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:58:16.512912 22014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:58:16.521587 22014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:58:16.521773 22014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:58:16.521952 22014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.130
I20260925 04:58:16.532632 22014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:34331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:45703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.130
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:58:16.534000 22014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:58:16.535380 22014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:58:16.549033 22019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:58:16.548954 22020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:58:16.549009 22022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:58:16.550010 22014 server_base.cc:1034] running on GCE node
I20260925 04:58:16.550673 22014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:58:16.551632 22014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:58:16.552867 22014 hybrid_clock.cc:648] HybridClock initialized: now 1790312296552811 us; error 41 us; skew 500 ppm
I20260925 04:58:16.553181 22014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:58:16 dist-test-slave-xzjr krb5kdc[21774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312296, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:58:16.556615 22014 init.cc:377] Logged in from keytab as kudu/127.6.29.130@KRBTEST.COM (short username kudu)
I20260925 04:58:16.558233 22014 webserver.cc:460] Webserver started at http://127.6.29.130:44231/ using document root <none> and password file <none>
I20260925 04:58:16.558974 22014 fs_manager.cc:359] Metadata directory not provided
I20260925 04:58:16.559079 22014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:58:16.559351 22014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:58:16.562644 22014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/data/instance:
uuid: "6ac80b17c4a341d09d85f9860bb22c11"
format_stamp: "Formatted at 2026-09-25 04:58:16 on dist-test-slave-xzjr"
server_key: "1b262c63405a21a238c14ccf0bd25c1a"
server_key_iv: "bf8f94fe78ea3a7dfda5ee070e478e63"
server_key_version: "encryptionkey@0"
I20260925 04:58:16.563413 22014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance:
uuid: "6ac80b17c4a341d09d85f9860bb22c11"
format_stamp: "Formatted at 2026-09-25 04:58:16 on dist-test-slave-xzjr"
server_key: "1b262c63405a21a238c14ccf0bd25c1a"
server_key_iv: "bf8f94fe78ea3a7dfda5ee070e478e63"
server_key_version: "encryptionkey@0"
I20260925 04:58:16.568969 22014 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 04:58:16.572770 22029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:58:16.574399 22014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260925 04:58:16.574590 22014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/wal
uuid: "6ac80b17c4a341d09d85f9860bb22c11"
format_stamp: "Formatted at 2026-09-25 04:58:16 on dist-test-slave-xzjr"
server_key: "1b262c63405a21a238c14ccf0bd25c1a"
server_key_iv: "bf8f94fe78ea3a7dfda5ee070e478e63"
server_key_version: "encryptionkey@0"
I20260925 04:58:16.574779 22014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:58:16.614037 22014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:58:16.618891 22014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:58:16.619203 22014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:58:16.620316 22014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:58:16.621799 22014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:58:16.621927 22014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:58:16.622076 22014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:58:16.622145 22014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:58:16.656116 22014 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.130:35999
I20260925 04:58:16.656163 22141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.130:35999 every 8 connection(s)
I20260925 04:58:16.657492 22014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
I20260925 04:58:16.658555 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 22014
I20260925 04:58:16.658731 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance
I20260925 04:58:16.659225 6262 external_mini_cluster.cc:1428] Setting key 310c06496a700b8812eb66e521f87630
Sep 25 04:58:16 dist-test-slave-xzjr krb5kdc[21774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312296, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:58:16.679965 22144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:16.660407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:54169 (local address 127.6.29.190:45703)
0925 04:58:16.660921 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0925 04:58:16.660927 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:58:16.661976 (+ 1049us) server_negotiation.cc:407] Connection header received
0925 04:58:16.663942 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:58:16.663947 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:58:16.664038 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:58:16.664158 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:58:16.665748 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:16.666504 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:16.668299 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:16.668873 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:16.668985 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:16.673450 (+ 4465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:58:16.673456 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:58:16.673459 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:58:16.673493 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:58:16.676466 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:58:16.677307 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:58:16.677312 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:58:16.677314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:58:16.677448 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:58:16.677924 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:58:16.677928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:58:16.677930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:58:16.678297 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:58:16.678474 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 04:58:16.679275 (+ 801us) server_negotiation.cc:299] Negotiation successful
0925 04:58:16.679520 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":902,"threads_started":1}
I20260925 04:58:16.681090 22145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:16.660992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.130:54169)
0925 04:58:16.661820 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:16.661871 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 04:58:16.663604 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:58:16.664430 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:58:16.664439 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:58:16.664967 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:58:16.665559 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:58:16.665580 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:58:16.666687 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:58:16.666692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:58:16.668113 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:58:16.668125 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:58:16.669128 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:58:16.669138 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:58:16.669497 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:16.670292 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:58:16.670380 (+ 88us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:58:16.673166 (+ 2786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:58:16.676636 (+ 3470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:58:16.676646 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:58:16.676674 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:58:16.677142 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:58:16.677570 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:58:16.677575 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:58:16.677579 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:58:16.677788 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:58:16.678441 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:58:16.678450 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:58:16.678877 (+ 427us) client_negotiation.cc:769] Sending connection context
0925 04:58:16.679163 (+ 286us) client_negotiation.cc:240] Negotiation successful
0925 04:58:16.679913 (+ 750us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":318,"threads_started":1}
I20260925 04:58:16.682790 22142 heartbeater.cc:347] Connected to a master server at 127.6.29.190:45703
I20260925 04:58:16.683130 22142 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.6.29.131@KRBTEST.COM; defaulting to no policy
I20260925 04:58:16.683990 22142 heartbeater.cc:511] Master 127.6.29.190:45703 requested a full tablet report, sending...
I20260925 04:58:16.685554 21819 ts_manager.cc:194] Registered new tserver with Master: 6ac80b17c4a341d09d85f9860bb22c11 (127.6.29.130:35999)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.131@KRBTEST.COM" created.
I20260925 04:58:16.686616 21819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.130@KRBTEST.COM'} at 127.6.29.130:54169
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab.
Entry for principal kudu/127.6.29.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab.
I20260925 04:58:16.717465 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.131:0
--local_ip_for_outbound_sockets=127.6.29.131
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:45703
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:34331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:58:17.061687 22148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:58:17.062039 22148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:58:17.062206 22148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:58:17.069617 22148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:58:17.069741 22148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:58:17.069912 22148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.131
I20260925 04:58:17.080500 22148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:34331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:45703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.131
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:58:17.081774 22148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:58:17.083120 22148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:58:17.097049 22156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:58:17.097151 22153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:58:17.097128 22154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:58:17.098026 22148 server_base.cc:1034] running on GCE node
I20260925 04:58:17.098716 22148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:58:17.099743 22148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:58:17.101025 22148 hybrid_clock.cc:648] HybridClock initialized: now 1790312297100984 us; error 31 us; skew 500 ppm
I20260925 04:58:17.101392 22148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:58:17 dist-test-slave-xzjr krb5kdc[21774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312297, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:58:17.104800 22148 init.cc:377] Logged in from keytab as kudu/127.6.29.131@KRBTEST.COM (short username kudu)
I20260925 04:58:17.106457 22148 webserver.cc:460] Webserver started at http://127.6.29.131:44155/ using document root <none> and password file <none>
I20260925 04:58:17.107195 22148 fs_manager.cc:359] Metadata directory not provided
I20260925 04:58:17.107347 22148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:58:17.107686 22148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:58:17.110769 22148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/data/instance:
uuid: "340b5359f50d4360b8e70651f02d5a66"
format_stamp: "Formatted at 2026-09-25 04:58:17 on dist-test-slave-xzjr"
server_key: "9ab89113e5201237eb9f12a1784906fb"
server_key_iv: "8115e489ea68979a8c2d89fa2098ed17"
server_key_version: "encryptionkey@0"
I20260925 04:58:17.111595 22148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance:
uuid: "340b5359f50d4360b8e70651f02d5a66"
format_stamp: "Formatted at 2026-09-25 04:58:17 on dist-test-slave-xzjr"
server_key: "9ab89113e5201237eb9f12a1784906fb"
server_key_iv: "8115e489ea68979a8c2d89fa2098ed17"
server_key_version: "encryptionkey@0"
I20260925 04:58:17.117213 22148 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 04:58:17.120944 22163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:58:17.121059 22008 heartbeater.cc:503] Master 127.6.29.190:45703 was elected leader, sending a full tablet report...
I20260925 04:58:17.122531 22148 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 04:58:17.122741 22148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/wal
uuid: "340b5359f50d4360b8e70651f02d5a66"
format_stamp: "Formatted at 2026-09-25 04:58:17 on dist-test-slave-xzjr"
server_key: "9ab89113e5201237eb9f12a1784906fb"
server_key_iv: "8115e489ea68979a8c2d89fa2098ed17"
server_key_version: "encryptionkey@0"
I20260925 04:58:17.122999 22148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:58:17.152825 22148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:58:17.157779 22148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:58:17.158104 22148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:58:17.159144 22148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:58:17.160559 22148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:58:17.160667 22148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:58:17.160840 22148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:58:17.160926 22148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:58:17.195358 22148 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.131:33855
I20260925 04:58:17.195453 22275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.131:33855 every 8 connection(s)
I20260925 04:58:17.196820 22148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
I20260925 04:58:17.202694 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 22148
I20260925 04:58:17.202883 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance
I20260925 04:58:17.203472 6262 external_mini_cluster.cc:1428] Setting key b092bb39cf0a381dc1b5388b52632cd1
Sep 25 04:58:17 dist-test-slave-xzjr krb5kdc[21774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312297, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:58:17.220929 22278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:17.199642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:42437 (local address 127.6.29.190:45703)
0925 04:58:17.200169 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0925 04:58:17.200173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:58:17.201108 (+ 935us) server_negotiation.cc:407] Connection header received
0925 04:58:17.203570 (+ 2462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:58:17.203580 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:58:17.203684 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:58:17.203859 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:58:17.205983 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:17.207288 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:17.209151 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:17.209618 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:17.209725 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:17.213966 (+ 4241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:58:17.213976 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:58:17.213984 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:58:17.214043 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:58:17.216953 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:58:17.218053 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:58:17.218060 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:58:17.218063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:58:17.218237 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:58:17.218850 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:58:17.218854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:58:17.218857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:58:17.219232 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:58:17.219468 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 04:58:17.220457 (+ 989us) server_negotiation.cc:299] Negotiation successful
0925 04:58:17.220638 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":298,"threads_started":1}
I20260925 04:58:17.222078 22279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:17.200206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.131:42437)
0925 04:58:17.200923 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:17.200971 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 04:58:17.203043 (+ 2072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:58:17.204149 (+ 1106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:58:17.204158 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:58:17.204883 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:58:17.205766 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:58:17.205795 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:58:17.207469 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:58:17.207474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:58:17.208939 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:58:17.208954 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:58:17.209844 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:58:17.209849 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:58:17.210070 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:17.210860 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:58:17.210884 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:58:17.213691 (+ 2807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:58:17.217125 (+ 3434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:58:17.217132 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:58:17.217147 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:58:17.217880 (+ 733us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:58:17.218395 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:58:17.218401 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:58:17.218405 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:58:17.218713 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:58:17.219412 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:58:17.219419 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:58:17.220038 (+ 619us) client_negotiation.cc:769] Sending connection context
0925 04:58:17.220351 (+ 313us) client_negotiation.cc:240] Negotiation successful
0925 04:58:17.220791 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":290,"threads_started":1}
I20260925 04:58:17.223644 22276 heartbeater.cc:347] Connected to a master server at 127.6.29.190:45703
I20260925 04:58:17.224048 22276 heartbeater.cc:464] Registering TS with master...
I20260925 04:58:17.224792 22276 heartbeater.cc:511] Master 127.6.29.190:45703 requested a full tablet report, sending...
I20260925 04:58:17.226341 21819 ts_manager.cc:194] Registered new tserver with Master: 340b5359f50d4360b8e70651f02d5a66 (127.6.29.131:33855)
I20260925 04:58:17.227299 21819 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.131@KRBTEST.COM'} at 127.6.29.131:42437
I20260925 04:58:17.233053 6262 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:58:17.252413 22278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:17.237826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44042 (local address 127.6.29.190:45703)
0925 04:58:17.238087 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0925 04:58:17.238095 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:58:17.238392 (+ 297us) server_negotiation.cc:407] Connection header received
0925 04:58:17.238572 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:58:17.238576 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:58:17.238644 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:58:17.238731 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:58:17.240257 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:17.241035 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:17.242385 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:17.243004 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:17.243130 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:17.246400 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:58:17.246405 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:58:17.246408 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:58:17.246449 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:58:17.249372 (+ 2923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:58:17.250188 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:58:17.250193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:58:17.250195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:58:17.250344 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:58:17.250719 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:58:17.250723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:58:17.250725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:58:17.251095 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:58:17.251363 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0925 04:58:17.251707 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 04:58:17.251987 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 04:58:17.256911 21819 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:58:17.264169 21819 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:58:17.301524 22291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:17.277512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46502 (local address 127.6.29.129:46341)
0925 04:58:17.291506 (+ 13994us) server_negotiation.cc:206] Beginning negotiation
0925 04:58:17.291521 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:58:17.292458 (+ 937us) server_negotiation.cc:407] Connection header received
0925 04:58:17.292898 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:58:17.292908 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:58:17.293198 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:58:17.293398 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:58:17.294599 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:17.296381 (+ 1782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:17.299130 (+ 2749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:17.300198 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:17.300635 (+ 437us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:17.300937 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0925 04:58:17.301009 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 04:58:17.301184 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12216,"thread_start_us":12535,"threads_started":1}
I20260925 04:58:17.301582 22289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:17.277432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:46341 (local address 127.0.0.1:46502)
0925 04:58:17.292261 (+ 14829us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:17.292305 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 04:58:17.292586 (+ 281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:58:17.293699 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:58:17.293729 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:58:17.293773 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:58:17.294364 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:58:17.294381 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:58:17.296543 (+ 2162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:58:17.296549 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:58:17.298929 (+ 2380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:58:17.298946 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:58:17.300456 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:58:17.300462 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:58:17.300730 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:17.300798 (+ 68us) client_negotiation.cc:769] Sending connection context
0925 04:58:17.300923 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 04:58:17.301142 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14533,"mutex_wait_us":110,"thread_start_us":15059,"threads_started":1}
I20260925 04:58:17.303568 22290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:17.277513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:35999 (local address 127.0.0.1:49322)
0925 04:58:17.292540 (+ 15027us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:17.292562 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 04:58:17.292704 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:58:17.295378 (+ 2674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:58:17.295383 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:58:17.295414 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:58:17.295754 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:58:17.295795 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:58:17.297717 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:58:17.297723 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:58:17.300167 (+ 2444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:58:17.300182 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:58:17.302710 (+ 2528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:58:17.302717 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:58:17.302922 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:17.302964 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 04:58:17.303046 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 04:58:17.303193 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14897,"mutex_wait_us":4,"thread_start_us":14714,"threads_started":1}
I20260925 04:58:17.304392 22292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:17.278060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49322 (local address 127.6.29.130:35999)
0925 04:58:17.292049 (+ 13989us) server_negotiation.cc:206] Beginning negotiation
0925 04:58:17.292057 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:58:17.292828 (+ 771us) server_negotiation.cc:407] Connection header received
0925 04:58:17.292944 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:58:17.292951 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:58:17.293203 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:58:17.295096 (+ 1893us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:58:17.295985 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:17.297564 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:17.300431 (+ 2867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:17.302499 (+ 2068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:17.302637 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:17.302831 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 04:58:17.303854 (+ 1023us) server_negotiation.cc:299] Negotiation successful
0925 04:58:17.304042 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13817,"thread_start_us":13638,"threads_started":1}
I20260925 04:58:17.305827 21943 tablet_service.cc:1467] Processing CreateTablet for tablet 9013201a70b543b08f0a1daf57722d80 (DEFAULT_TABLE table=test-table [id=c1c76bfa1f6849658f242bfefe89d83a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:58:17.307291 22293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:17.293492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.131:33855 (local address 127.0.0.1:55130)
0925 04:58:17.294973 (+ 1481us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:17.294992 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 04:58:17.295097 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:58:17.296411 (+ 1314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:58:17.296418 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:58:17.296447 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:58:17.296924 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:58:17.296933 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:58:17.299025 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:58:17.299030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:58:17.301085 (+ 2055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:58:17.301097 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:58:17.302960 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:58:17.302965 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:58:17.306632 (+ 3667us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:17.306666 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 04:58:17.306746 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 04:58:17.306907 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1376,"mutex_wait_us":1,"thread_start_us":1231,"threads_started":1}
I20260925 04:58:17.307318 21943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9013201a70b543b08f0a1daf57722d80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:58:17.308509 22294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:17.293629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55130 (local address 127.6.29.131:33855)
0925 04:58:17.295690 (+ 2061us) server_negotiation.cc:206] Beginning negotiation
0925 04:58:17.295700 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:58:17.295803 (+ 103us) server_negotiation.cc:407] Connection header received
0925 04:58:17.295916 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:58:17.295925 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:58:17.296197 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:58:17.297085 (+ 888us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:58:17.297328 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:17.298851 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:17.301251 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:17.302740 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:17.307583 (+ 4843us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:17.307860 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0925 04:58:17.307949 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 04:58:17.308132 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1872,"thread_start_us":1686,"threads_started":1}
I20260925 04:58:17.307989 22077 tablet_service.cc:1467] Processing CreateTablet for tablet 9013201a70b543b08f0a1daf57722d80 (DEFAULT_TABLE table=test-table [id=c1c76bfa1f6849658f242bfefe89d83a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:58:17.310276 22077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9013201a70b543b08f0a1daf57722d80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:58:17.312229 22211 tablet_service.cc:1467] Processing CreateTablet for tablet 9013201a70b543b08f0a1daf57722d80 (DEFAULT_TABLE table=test-table [id=c1c76bfa1f6849658f242bfefe89d83a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:58:17.313787 22211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9013201a70b543b08f0a1daf57722d80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:58:17.319976 22295 tablet_bootstrap.cc:492] T 9013201a70b543b08f0a1daf57722d80 P 7071476802644c14854c4de13332b96f: Bootstrap starting.
I20260925 04:58:17.323276 22295 tablet_bootstrap.cc:654] T 9013201a70b543b08f0a1daf57722d80 P 7071476802644c14854c4de13332b96f: Neither blocks nor log segments found. Creating new log.
I20260925 04:58:17.324458 22295 log.cc:824] T 9013201a70b543b08f0a1daf57722d80 P 7071476802644c14854c4de13332b96f: Log is configured to *not* fsync() on all Append() calls
I20260925 04:58:17.327921 22297 tablet_bootstrap.cc:492] T 9013201a70b543b08f0a1daf57722d80 P 6ac80b17c4a341d09d85f9860bb22c11: Bootstrap starting.
I20260925 04:58:17.331795 22295 tablet_bootstrap.cc:492] T 9013201a70b543b08f0a1daf57722d80 P 7071476802644c14854c4de13332b96f: No bootstrap required, opened a new log
I20260925 04:58:17.331938 22298 tablet_bootstrap.cc:492] T 9013201a70b543b08f0a1daf57722d80 P 340b5359f50d4360b8e70651f02d5a66: Bootstrap starting.
I20260925 04:58:17.332114 22295 ts_tablet_manager.cc:1397] T 9013201a70b543b08f0a1daf57722d80 P 7071476802644c14854c4de13332b96f: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.005s
I20260925 04:58:17.332113 22297 tablet_bootstrap.cc:654] T 9013201a70b543b08f0a1daf57722d80 P 6ac80b17c4a341d09d85f9860bb22c11: Neither blocks nor log segments found. Creating new log.
I20260925 04:58:17.333437 22297 log.cc:824] T 9013201a70b543b08f0a1daf57722d80 P 6ac80b17c4a341d09d85f9860bb22c11: Log is configured to *not* fsync() on all Append() calls
I20260925 04:58:17.335335 22298 tablet_bootstrap.cc:654] T 9013201a70b543b08f0a1daf57722d80 P 340b5359f50d4360b8e70651f02d5a66: Neither blocks nor log segments found. Creating new log.
I20260925 04:58:17.336776 22298 log.cc:824] T 9013201a70b543b08f0a1daf57722d80 P 340b5359f50d4360b8e70651f02d5a66: Log is configured to *not* fsync() on all Append() calls
I20260925 04:58:17.337008 22297 tablet_bootstrap.cc:492] T 9013201a70b543b08f0a1daf57722d80 P 6ac80b17c4a341d09d85f9860bb22c11: No bootstrap required, opened a new log
I20260925 04:58:17.337298 22297 ts_tablet_manager.cc:1397] T 9013201a70b543b08f0a1daf57722d80 P 6ac80b17c4a341d09d85f9860bb22c11: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260925 04:58:17.340194 22298 tablet_bootstrap.cc:492] T 9013201a70b543b08f0a1daf57722d80 P 340b5359f50d4360b8e70651f02d5a66: No bootstrap required, opened a new log
I20260925 04:58:17.340489 22298 ts_tablet_manager.cc:1397] T 9013201a70b543b08f0a1daf57722d80 P 340b5359f50d4360b8e70651f02d5a66: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260925 04:58:17.342593 22295 raft_consensus.cc:348] T 9013201a70b543b08f0a1daf57722d80 P 7071476802644c14854c4de13332b96f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7071476802644c14854c4de13332b96f" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 46341 } } peers { permanent_uuid: "6ac80b17c4a341d09d85f9860bb22c11" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35999 } } peers { permanent_uuid: "340b5359f50d4360b8e70651f02d5a66" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 33855 } }
I20260925 04:58:17.343073 22295 raft_consensus.cc:374] T 9013201a70b543b08f0a1daf57722d80 P 7071476802644c14854c4de13332b96f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:58:17.343174 22295 raft_consensus.cc:729] T 9013201a70b543b08f0a1daf57722d80 P 7071476802644c14854c4de13332b96f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7071476802644c14854c4de13332b96f, State: Initialized, Role: FOLLOWER
I20260925 04:58:17.343876 22295 consensus_queue.cc:260] T 9013201a70b543b08f0a1daf57722d80 P 7071476802644c14854c4de13332b96f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7071476802644c14854c4de13332b96f" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 46341 } } peers { permanent_uuid: "6ac80b17c4a341d09d85f9860bb22c11" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35999 } } peers { permanent_uuid: "340b5359f50d4360b8e70651f02d5a66" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 33855 } }
I20260925 04:58:17.345523 22297 raft_consensus.cc:348] T 9013201a70b543b08f0a1daf57722d80 P 6ac80b17c4a341d09d85f9860bb22c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7071476802644c14854c4de13332b96f" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 46341 } } peers { permanent_uuid: "6ac80b17c4a341d09d85f9860bb22c11" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35999 } } peers { permanent_uuid: "340b5359f50d4360b8e70651f02d5a66" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 33855 } }
I20260925 04:58:17.345965 22297 raft_consensus.cc:374] T 9013201a70b543b08f0a1daf57722d80 P 6ac80b17c4a341d09d85f9860bb22c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:58:17.345994 22295 ts_tablet_manager.cc:1428] T 9013201a70b543b08f0a1daf57722d80 P 7071476802644c14854c4de13332b96f: Time spent starting tablet: real 0.014s user 0.011s sys 0.002s
I20260925 04:58:17.346056 22297 raft_consensus.cc:729] T 9013201a70b543b08f0a1daf57722d80 P 6ac80b17c4a341d09d85f9860bb22c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ac80b17c4a341d09d85f9860bb22c11, State: Initialized, Role: FOLLOWER
I20260925 04:58:17.346678 22297 consensus_queue.cc:260] T 9013201a70b543b08f0a1daf57722d80 P 6ac80b17c4a341d09d85f9860bb22c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7071476802644c14854c4de13332b96f" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 46341 } } peers { permanent_uuid: "6ac80b17c4a341d09d85f9860bb22c11" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35999 } } peers { permanent_uuid: "340b5359f50d4360b8e70651f02d5a66" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 33855 } }
I20260925 04:58:17.347886 22142 heartbeater.cc:503] Master 127.6.29.190:45703 was elected leader, sending a full tablet report...
I20260925 04:58:17.348528 22297 ts_tablet_manager.cc:1428] T 9013201a70b543b08f0a1daf57722d80 P 6ac80b17c4a341d09d85f9860bb22c11: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260925 04:58:17.351969 22298 raft_consensus.cc:348] T 9013201a70b543b08f0a1daf57722d80 P 340b5359f50d4360b8e70651f02d5a66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7071476802644c14854c4de13332b96f" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 46341 } } peers { permanent_uuid: "6ac80b17c4a341d09d85f9860bb22c11" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35999 } } peers { permanent_uuid: "340b5359f50d4360b8e70651f02d5a66" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 33855 } }
I20260925 04:58:17.352365 22298 raft_consensus.cc:374] T 9013201a70b543b08f0a1daf57722d80 P 340b5359f50d4360b8e70651f02d5a66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:58:17.352488 22298 raft_consensus.cc:729] T 9013201a70b543b08f0a1daf57722d80 P 340b5359f50d4360b8e70651f02d5a66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 340b5359f50d4360b8e70651f02d5a66, State: Initialized, Role: FOLLOWER
I20260925 04:58:17.353135 22298 consensus_queue.cc:260] T 9013201a70b543b08f0a1daf57722d80 P 340b5359f50d4360b8e70651f02d5a66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7071476802644c14854c4de13332b96f" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 46341 } } peers { permanent_uuid: "6ac80b17c4a341d09d85f9860bb22c11" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35999 } } peers { permanent_uuid: "340b5359f50d4360b8e70651f02d5a66" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 33855 } }
I20260925 04:58:17.354287 22276 heartbeater.cc:503] Master 127.6.29.190:45703 was elected leader, sending a full tablet report...
I20260925 04:58:17.354938 22298 ts_tablet_manager.cc:1428] T 9013201a70b543b08f0a1daf57722d80 P 340b5359f50d4360b8e70651f02d5a66: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260925 04:58:17.409776 22143 tablet.cc:2369] T 9013201a70b543b08f0a1daf57722d80 P 6ac80b17c4a341d09d85f9860bb22c11: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:58:17.448586 22277 tablet.cc:2369] T 9013201a70b543b08f0a1daf57722d80 P 340b5359f50d4360b8e70651f02d5a66: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:58:17.451627 22303 raft_consensus.cc:482] T 9013201a70b543b08f0a1daf57722d80 P 340b5359f50d4360b8e70651f02d5a66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:58:17.451980 22303 raft_consensus.cc:504] T 9013201a70b543b08f0a1daf57722d80 P 340b5359f50d4360b8e70651f02d5a66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7071476802644c14854c4de13332b96f" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 46341 } } peers { permanent_uuid: "6ac80b17c4a341d09d85f9860bb22c11" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35999 } } peers { permanent_uuid: "340b5359f50d4360b8e70651f02d5a66" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 33855 } }
I20260925 04:58:17.453698 22303 leader_election.cc:288] T 9013201a70b543b08f0a1daf57722d80 P 340b5359f50d4360b8e70651f02d5a66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7071476802644c14854c4de13332b96f (127.6.29.129:46341), 6ac80b17c4a341d09d85f9860bb22c11 (127.6.29.130:35999)
I20260925 04:58:17.460816 22292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:17.454333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:57137 (local address 127.6.29.130:35999)
0925 04:58:17.454541 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 04:58:17.454550 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:58:17.454711 (+ 161us) server_negotiation.cc:407] Connection header received
0925 04:58:17.454917 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:58:17.454926 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:58:17.455033 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:58:17.455135 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:58:17.455622 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:17.456656 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:17.458618 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:17.459927 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:17.460052 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:17.460107 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 04:58:17.460490 (+ 383us) server_negotiation.cc:299] Negotiation successful
0925 04:58:17.460593 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 04:58:17.460979 22279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:17.454336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:35999 (local address 127.6.29.131:57137)
0925 04:58:17.454596 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:17.454616 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 04:58:17.454745 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:58:17.455178 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:58:17.455185 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:58:17.455225 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:58:17.455446 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:58:17.455462 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:58:17.456784 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:58:17.456790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:58:17.458451 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:58:17.458464 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:58:17.460082 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:58:17.460093 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:58:17.460308 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:17.460345 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 04:58:17.460465 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 04:58:17.460616 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260925 04:58:17.462352 22304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:17.454339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:46341 (local address 127.6.29.131:54377)
0925 04:58:17.455363 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:17.455395 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 04:58:17.455537 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:58:17.455883 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:58:17.455890 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:58:17.455914 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:58:17.456279 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:58:17.456290 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:58:17.457683 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:58:17.457686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:58:17.459514 (+ 1828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:58:17.459531 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:58:17.461492 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:58:17.461498 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:58:17.461685 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:17.461721 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 04:58:17.461803 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 04:58:17.461901 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"mutex_wait_us":215,"thread_start_us":422,"threads_started":1}
I20260925 04:58:17.462345 22291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:17.454277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:54377 (local address 127.6.29.129:46341)
0925 04:58:17.454542 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0925 04:58:17.454548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:58:17.455559 (+ 1011us) server_negotiation.cc:407] Connection header received
0925 04:58:17.455649 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:58:17.455653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:58:17.455714 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:58:17.455873 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:58:17.456446 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:17.457525 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:17.459721 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:17.461079 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:17.461273 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:17.461357 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 04:58:17.461921 (+ 564us) server_negotiation.cc:299] Negotiation successful
0925 04:58:17.462099 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260925 04:58:17.462368 22097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9013201a70b543b08f0a1daf57722d80" candidate_uuid: "340b5359f50d4360b8e70651f02d5a66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ac80b17c4a341d09d85f9860bb22c11" is_pre_election: true
I20260925 04:58:17.463093 22097 raft_consensus.cc:2454] T 9013201a70b543b08f0a1daf57722d80 P 6ac80b17c4a341d09d85f9860bb22c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 340b5359f50d4360b8e70651f02d5a66 in term 0.
I20260925 04:58:17.463994 22167 leader_election.cc:302] T 9013201a70b543b08f0a1daf57722d80 P 340b5359f50d4360b8e70651f02d5a66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 340b5359f50d4360b8e70651f02d5a66, 6ac80b17c4a341d09d85f9860bb22c11; no voters:
I20260925 04:58:17.463969 21963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9013201a70b543b08f0a1daf57722d80" candidate_uuid: "340b5359f50d4360b8e70651f02d5a66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7071476802644c14854c4de13332b96f" is_pre_election: true
I20260925 04:58:17.464406 22303 raft_consensus.cc:2781] T 9013201a70b543b08f0a1daf57722d80 P 340b5359f50d4360b8e70651f02d5a66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:58:17.464548 22303 raft_consensus.cc:482] T 9013201a70b543b08f0a1daf57722d80 P 340b5359f50d4360b8e70651f02d5a66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:58:17.464665 22303 raft_consensus.cc:3037] T 9013201a70b543b08f0a1daf57722d80 P 340b5359f50d4360b8e70651f02d5a66 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:58:17.464596 21963 raft_consensus.cc:2454] T 9013201a70b543b08f0a1daf57722d80 P 7071476802644c14854c4de13332b96f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 340b5359f50d4360b8e70651f02d5a66 in term 0.
I20260925 04:58:17.467264 22303 raft_consensus.cc:504] T 9013201a70b543b08f0a1daf57722d80 P 340b5359f50d4360b8e70651f02d5a66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7071476802644c14854c4de13332b96f" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 46341 } } peers { permanent_uuid: "6ac80b17c4a341d09d85f9860bb22c11" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35999 } } peers { permanent_uuid: "340b5359f50d4360b8e70651f02d5a66" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 33855 } }
I20260925 04:58:17.467998 22303 leader_election.cc:288] T 9013201a70b543b08f0a1daf57722d80 P 340b5359f50d4360b8e70651f02d5a66 [CANDIDATE]: Term 1 election: Requested vote from peers 7071476802644c14854c4de13332b96f (127.6.29.129:46341), 6ac80b17c4a341d09d85f9860bb22c11 (127.6.29.130:35999)
I20260925 04:58:17.468498 21963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9013201a70b543b08f0a1daf57722d80" candidate_uuid: "340b5359f50d4360b8e70651f02d5a66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7071476802644c14854c4de13332b96f"
I20260925 04:58:17.468535 22097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9013201a70b543b08f0a1daf57722d80" candidate_uuid: "340b5359f50d4360b8e70651f02d5a66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ac80b17c4a341d09d85f9860bb22c11"
I20260925 04:58:17.468729 21963 raft_consensus.cc:3037] T 9013201a70b543b08f0a1daf57722d80 P 7071476802644c14854c4de13332b96f [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:58:17.468736 22097 raft_consensus.cc:3037] T 9013201a70b543b08f0a1daf57722d80 P 6ac80b17c4a341d09d85f9860bb22c11 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:58:17.470927 22097 raft_consensus.cc:2454] T 9013201a70b543b08f0a1daf57722d80 P 6ac80b17c4a341d09d85f9860bb22c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 340b5359f50d4360b8e70651f02d5a66 in term 1.
I20260925 04:58:17.470947 21963 raft_consensus.cc:2454] T 9013201a70b543b08f0a1daf57722d80 P 7071476802644c14854c4de13332b96f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 340b5359f50d4360b8e70651f02d5a66 in term 1.
I20260925 04:58:17.471377 22167 leader_election.cc:302] T 9013201a70b543b08f0a1daf57722d80 P 340b5359f50d4360b8e70651f02d5a66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 340b5359f50d4360b8e70651f02d5a66, 6ac80b17c4a341d09d85f9860bb22c11; no voters:
I20260925 04:58:17.471700 22303 raft_consensus.cc:2781] T 9013201a70b543b08f0a1daf57722d80 P 340b5359f50d4360b8e70651f02d5a66 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:58:17.472186 22303 raft_consensus.cc:686] T 9013201a70b543b08f0a1daf57722d80 P 340b5359f50d4360b8e70651f02d5a66 [term 1 LEADER]: Becoming Leader. State: Replica: 340b5359f50d4360b8e70651f02d5a66, State: Running, Role: LEADER
I20260925 04:58:17.472793 22303 consensus_queue.cc:237] T 9013201a70b543b08f0a1daf57722d80 P 340b5359f50d4360b8e70651f02d5a66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7071476802644c14854c4de13332b96f" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 46341 } } peers { permanent_uuid: "6ac80b17c4a341d09d85f9860bb22c11" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35999 } } peers { permanent_uuid: "340b5359f50d4360b8e70651f02d5a66" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 33855 } }
I20260925 04:58:17.477713 21818 catalog_manager.cc:5495] T 9013201a70b543b08f0a1daf57722d80 P 340b5359f50d4360b8e70651f02d5a66 reported cstate change: term changed from 0 to 1, leader changed from <none> to 340b5359f50d4360b8e70651f02d5a66 (127.6.29.131). New cstate: current_term: 1 leader_uuid: "340b5359f50d4360b8e70651f02d5a66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7071476802644c14854c4de13332b96f" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 46341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ac80b17c4a341d09d85f9860bb22c11" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "340b5359f50d4360b8e70651f02d5a66" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 33855 } health_report { overall_health: HEALTHY } } }
I20260925 04:58:17.501324 22294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:17.495579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55144 (local address 127.6.29.131:33855)
0925 04:58:17.495836 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0925 04:58:17.495841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:58:17.495860 (+ 19us) server_negotiation.cc:407] Connection header received
0925 04:58:17.495982 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:58:17.495986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:58:17.496063 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:58:17.496193 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:58:17.496585 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:17.497333 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:17.498916 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:17.499347 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:17.499445 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:17.500047 (+ 602us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:58:17.500642 (+ 595us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:58:17.500765 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 04:58:17.500886 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 04:58:17.500962 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 04:58:17.510306 21963 raft_consensus.cc:1261] T 9013201a70b543b08f0a1daf57722d80 P 7071476802644c14854c4de13332b96f [term 1 FOLLOWER]: Refusing update from remote peer 340b5359f50d4360b8e70651f02d5a66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:58:17.510330 22097 raft_consensus.cc:1261] T 9013201a70b543b08f0a1daf57722d80 P 6ac80b17c4a341d09d85f9860bb22c11 [term 1 FOLLOWER]: Refusing update from remote peer 340b5359f50d4360b8e70651f02d5a66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:58:17.511363 22305 consensus_queue.cc:1035] T 9013201a70b543b08f0a1daf57722d80 P 340b5359f50d4360b8e70651f02d5a66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7071476802644c14854c4de13332b96f" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 46341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:58:17.511809 22303 consensus_queue.cc:1035] T 9013201a70b543b08f0a1daf57722d80 P 340b5359f50d4360b8e70651f02d5a66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ac80b17c4a341d09d85f9860bb22c11" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:58:17.540745 21819 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:58:17.541029 21819 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:58:17.544745 21819 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b: Sending DeleteTablet for 3 replicas of tablet 9013201a70b543b08f0a1daf57722d80
I20260925 04:58:17.546082 22077 tablet_service.cc:1514] Processing DeleteTablet for tablet 9013201a70b543b08f0a1daf57722d80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:58:17 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:49322
I20260925 04:58:17.546156 22211 tablet_service.cc:1514] Processing DeleteTablet for tablet 9013201a70b543b08f0a1daf57722d80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:58:17 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:55130
I20260925 04:58:17.546134 21943 tablet_service.cc:1514] Processing DeleteTablet for tablet 9013201a70b543b08f0a1daf57722d80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:58:17 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:46502
I20260925 04:58:17.546640 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 21880
I20260925 04:58:17.558924 22318 tablet_replica.cc:330] T 9013201a70b543b08f0a1daf57722d80 P 340b5359f50d4360b8e70651f02d5a66: stopping tablet replica
I20260925 04:58:17.559695 22320 tablet_replica.cc:330] T 9013201a70b543b08f0a1daf57722d80 P 7071476802644c14854c4de13332b96f: stopping tablet replica
I20260925 04:58:17.560585 22320 raft_consensus.cc:2229] T 9013201a70b543b08f0a1daf57722d80 P 7071476802644c14854c4de13332b96f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:58:17.561188 22318 raft_consensus.cc:2229] T 9013201a70b543b08f0a1daf57722d80 P 340b5359f50d4360b8e70651f02d5a66 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:58:17.561275 22319 tablet_replica.cc:330] T 9013201a70b543b08f0a1daf57722d80 P 6ac80b17c4a341d09d85f9860bb22c11: stopping tablet replica
I20260925 04:58:17.561664 22319 raft_consensus.cc:2229] T 9013201a70b543b08f0a1daf57722d80 P 6ac80b17c4a341d09d85f9860bb22c11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:58:17.562237 22319 raft_consensus.cc:2258] T 9013201a70b543b08f0a1daf57722d80 P 6ac80b17c4a341d09d85f9860bb22c11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:58:17.562237 22320 raft_consensus.cc:2258] T 9013201a70b543b08f0a1daf57722d80 P 7071476802644c14854c4de13332b96f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:58:17.562604 22318 raft_consensus.cc:2258] T 9013201a70b543b08f0a1daf57722d80 P 340b5359f50d4360b8e70651f02d5a66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:58:17.565106 22319 ts_tablet_manager.cc:1905] T 9013201a70b543b08f0a1daf57722d80 P 6ac80b17c4a341d09d85f9860bb22c11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:58:17.567111 22320 ts_tablet_manager.cc:1905] T 9013201a70b543b08f0a1daf57722d80 P 7071476802644c14854c4de13332b96f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:58:17.568048 22318 ts_tablet_manager.cc:1905] T 9013201a70b543b08f0a1daf57722d80 P 340b5359f50d4360b8e70651f02d5a66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:58:17.572435 22319 ts_tablet_manager.cc:1918] T 9013201a70b543b08f0a1daf57722d80 P 6ac80b17c4a341d09d85f9860bb22c11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:58:17.572634 22319 log.cc:1196] T 9013201a70b543b08f0a1daf57722d80 P 6ac80b17c4a341d09d85f9860bb22c11: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/wal/wals/9013201a70b543b08f0a1daf57722d80
Sep 25 04:58:17 dist-test-slave-xzjr krb5kdc[21774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.129@KRBTEST.COM
I20260925 04:58:17.573202 22319 ts_tablet_manager.cc:1939] T 9013201a70b543b08f0a1daf57722d80 P 6ac80b17c4a341d09d85f9860bb22c11: Deleting consensus metadata
I20260925 04:58:17.575248 21806 catalog_manager.cc:4841] TS 6ac80b17c4a341d09d85f9860bb22c11 (127.6.29.130:35999): tablet 9013201a70b543b08f0a1daf57722d80 (table test-table [id=c1c76bfa1f6849658f242bfefe89d83a]) successfully deleted
I20260925 04:58:17.579094 22320 ts_tablet_manager.cc:1918] T 9013201a70b543b08f0a1daf57722d80 P 7071476802644c14854c4de13332b96f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:58:17.579358 22320 log.cc:1196] T 9013201a70b543b08f0a1daf57722d80 P 7071476802644c14854c4de13332b96f: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/wal/wals/9013201a70b543b08f0a1daf57722d80
I20260925 04:58:17.579916 22320 ts_tablet_manager.cc:1939] T 9013201a70b543b08f0a1daf57722d80 P 7071476802644c14854c4de13332b96f: Deleting consensus metadata
I20260925 04:58:17.582669 21804 catalog_manager.cc:4841] TS 7071476802644c14854c4de13332b96f (127.6.29.129:46341): tablet 9013201a70b543b08f0a1daf57722d80 (table test-table [id=c1c76bfa1f6849658f242bfefe89d83a]) successfully deleted
I20260925 04:58:17.583338 22291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:17.548029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46512 (local address 127.6.29.129:46341)
0925 04:58:17.548268 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 04:58:17.548277 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:58:17.560447 (+ 12170us) server_negotiation.cc:407] Connection header received
0925 04:58:17.561026 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:58:17.561035 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:58:17.561137 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:58:17.561222 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:58:17.563621 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:17.564855 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:17.567101 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:17.567821 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:17.567947 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:17.574122 (+ 6175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:58:17.574146 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:58:17.574172 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:58:17.574235 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:58:17.579115 (+ 4880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:58:17.580203 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:58:17.580217 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:58:17.580224 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:58:17.580413 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:58:17.580941 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:58:17.580948 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:58:17.580951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:58:17.581606 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:58:17.581934 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0925 04:58:17.582598 (+ 664us) server_negotiation.cc:299] Negotiation successful
0925 04:58:17.582958 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 04:58:17.584360 22318 ts_tablet_manager.cc:1918] T 9013201a70b543b08f0a1daf57722d80 P 340b5359f50d4360b8e70651f02d5a66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:58:17.584425 22003 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:58:17.584575 22318 log.cc:1196] T 9013201a70b543b08f0a1daf57722d80 P 340b5359f50d4360b8e70651f02d5a66: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/wal/wals/9013201a70b543b08f0a1daf57722d80
I20260925 04:58:17.585172 22318 ts_tablet_manager.cc:1939] T 9013201a70b543b08f0a1daf57722d80 P 340b5359f50d4360b8e70651f02d5a66: Deleting consensus metadata
I20260925 04:58:17.587270 21804 catalog_manager.cc:4841] TS 340b5359f50d4360b8e70651f02d5a66 (127.6.29.131:33855): tablet 9013201a70b543b08f0a1daf57722d80 (table test-table [id=c1c76bfa1f6849658f242bfefe89d83a]) successfully deleted
I20260925 04:58:17.725041 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 21880
I20260925 04:58:17.744303 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 22014
Sep 25 04:58:17 dist-test-slave-xzjr krb5kdc[21774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.130@KRBTEST.COM
I20260925 04:58:17.764098 22292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:17.745195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49330 (local address 127.6.29.130:35999)
0925 04:58:17.745401 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 04:58:17.745405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:58:17.745486 (+ 81us) server_negotiation.cc:407] Connection header received
0925 04:58:17.745658 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:58:17.745662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:58:17.745761 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:58:17.745848 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:58:17.747638 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:17.748464 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:17.750485 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:17.750904 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:17.751019 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:17.756685 (+ 5666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:58:17.756710 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:58:17.756734 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:58:17.756789 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:58:17.760914 (+ 4125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:58:17.761717 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:58:17.761727 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:58:17.761731 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:58:17.761892 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:58:17.762274 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:58:17.762279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:58:17.762283 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:58:17.762791 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:58:17.763108 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0925 04:58:17.763446 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 04:58:17.763682 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 04:58:17.764947 22137 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:58:17.858480 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 22014
I20260925 04:58:17.876998 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 22148
Sep 25 04:58:17 dist-test-slave-xzjr krb5kdc[21774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.131@KRBTEST.COM
I20260925 04:58:17.897321 22294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:17.877824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55148 (local address 127.6.29.131:33855)
0925 04:58:17.878006 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 04:58:17.878014 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:58:17.878035 (+ 21us) server_negotiation.cc:407] Connection header received
0925 04:58:17.878372 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:58:17.878379 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:58:17.878480 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:58:17.878593 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:58:17.880482 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:17.881737 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:17.883315 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:17.884138 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:17.884287 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:17.889280 (+ 4993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:58:17.889297 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:58:17.889318 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:58:17.889390 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:58:17.893790 (+ 4400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:58:17.894738 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:58:17.894751 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:58:17.894757 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:58:17.894964 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:58:17.895421 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:58:17.895431 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:58:17.895436 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:58:17.896010 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:58:17.896344 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0925 04:58:17.896636 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 04:58:17.896915 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 04:58:17.898219 22271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:58:18.028879 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 22148
I20260925 04:58:18.052074 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 21787
I20260925 04:58:18.053294 21849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:58:18.166162 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 21787
I20260925 04:58:38.188406 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:45703
--webserver_interface=127.6.29.190
--webserver_port=46627
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:34331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:45703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:58:38.533674 22326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:58:38.533965 22326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:58:38.534126 22326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:58:38.541679 22326 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 04:58:38.541779 22326 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 04:58:38.541842 22326 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 04:58:38.541917 22326 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:58:38.542014 22326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:58:38.542135 22326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:58:38.542215 22326 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:58:38.542299 22326 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:58:38.552645 22326 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:34331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:45703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:45703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=46627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:58:38.554011 22326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:58:38.555449 22326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:58:38.579133 22333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:58:38.579289 22335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:58:38.579407 22332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:58:38.579938 22326 server_base.cc:1034] running on GCE node
I20260925 04:58:38.580925 22326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:58:38.582593 22326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:58:38.583873 22326 hybrid_clock.cc:648] HybridClock initialized: now 1790312318583797 us; error 53 us; skew 500 ppm
I20260925 04:58:38.584282 22326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:58:38 dist-test-slave-xzjr krb5kdc[21774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312318, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:58:38.592437 22326 init.cc:377] Logged in from keytab as kudu/127.6.29.190@KRBTEST.COM (short username kudu)
I20260925 04:58:38.594175 22326 webserver.cc:460] Webserver started at http://127.6.29.190:46627/ using document root <none> and password file <none>
I20260925 04:58:38.594954 22326 fs_manager.cc:359] Metadata directory not provided
I20260925 04:58:38.595055 22326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:58:38.601526 22326 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:58:38.604710 22342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:58:38.606194 22326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:58:38.606407 22326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "c42d92ab26a24f459a6f7dd1b623038b"
format_stamp: "Formatted at 2026-09-25 04:58:15 on dist-test-slave-xzjr"
server_key: "c9b22eb5243941d99b53fa7fd6ed3e91"
server_key_iv: "12720e2e4b5487fbb25b272d43b033e0"
server_key_version: "encryptionkey@0"
I20260925 04:58:38.607501 22326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:58:38.622498 22326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:58:38.627175 22326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:58:38.627434 22326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:58:38.648241 22326 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:45703
I20260925 04:58:38.648263 22393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:45703 every 8 connection(s)
I20260925 04:58:38.649942 22326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:58:38.653470 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 22326
I20260925 04:58:38.654331 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.129:46341
--local_ip_for_outbound_sockets=127.6.29.129
--tserver_master_addrs=127.6.29.190:45703
--webserver_port=40339
--webserver_interface=127.6.29.129
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:34331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:58:38.657847 22394 sys_catalog.cc:263] Verifying existing consensus state
I20260925 04:58:38.661377 22394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b: Bootstrap starting.
I20260925 04:58:38.681819 22394 log.cc:824] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b: Log is configured to *not* fsync() on all Append() calls
I20260925 04:58:38.694118 22394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:58:38.694696 22394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b: Bootstrap complete.
I20260925 04:58:38.704833 22394 raft_consensus.cc:348] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42d92ab26a24f459a6f7dd1b623038b" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 45703 } }
I20260925 04:58:38.705711 22394 raft_consensus.cc:729] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c42d92ab26a24f459a6f7dd1b623038b, State: Initialized, Role: FOLLOWER
I20260925 04:58:38.706418 22394 consensus_queue.cc:260] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42d92ab26a24f459a6f7dd1b623038b" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 45703 } }
I20260925 04:58:38.706794 22394 raft_consensus.cc:388] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:58:38.707006 22394 raft_consensus.cc:482] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:58:38.707208 22394 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [term 1 FOLLOWER]: Advancing to term 2
I20260925 04:58:38.711553 22394 raft_consensus.cc:504] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42d92ab26a24f459a6f7dd1b623038b" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 45703 } }
I20260925 04:58:38.712234 22394 leader_election.cc:302] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c42d92ab26a24f459a6f7dd1b623038b; no voters:
I20260925 04:58:38.712944 22394 leader_election.cc:288] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 04:58:38.713096 22398 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [term 2 FOLLOWER]: Leader election won for term 2
I20260925 04:58:38.715494 22394 sys_catalog.cc:564] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:58:38.716053 22398 raft_consensus.cc:686] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [term 2 LEADER]: Becoming Leader. State: Replica: c42d92ab26a24f459a6f7dd1b623038b, State: Running, Role: LEADER
I20260925 04:58:38.716701 22398 consensus_queue.cc:237] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42d92ab26a24f459a6f7dd1b623038b" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 45703 } }
I20260925 04:58:38.719942 22399 sys_catalog.cc:455] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c42d92ab26a24f459a6f7dd1b623038b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42d92ab26a24f459a6f7dd1b623038b" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 45703 } } }
I20260925 04:58:38.720288 22399 sys_catalog.cc:458] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [sys.catalog]: This master's current role is: LEADER
I20260925 04:58:38.719460 22400 sys_catalog.cc:455] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c42d92ab26a24f459a6f7dd1b623038b. Latest consensus state: current_term: 2 leader_uuid: "c42d92ab26a24f459a6f7dd1b623038b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42d92ab26a24f459a6f7dd1b623038b" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 45703 } } }
I20260925 04:58:38.720858 22400 sys_catalog.cc:458] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [sys.catalog]: This master's current role is: LEADER
I20260925 04:58:38.729431 22405 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:58:38.740662 22405 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:58:38.742621 22405 catalog_manager.cc:1259] Loaded cluster ID: 23f56071d0524962a48d3f716d58e81a
I20260925 04:58:38.742848 22405 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:58:38.747311 22415 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 04:58:38.748063 22405 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:58:38.749209 22405 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b: Loaded TSK: 0
I20260925 04:58:38.774058 22405 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b: Generated new TSK 1
I20260925 04:58:38.774483 22405 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 04:58:39.044818 22396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:58:39.045118 22396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:58:39.045287 22396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:58:39.053124 22396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:58:39.053259 22396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:58:39.053432 22396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.129
I20260925 04:58:39.064366 22396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:34331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.129:46341
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.29.129
--webserver_port=40339
--tserver_master_addrs=127.6.29.190:45703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.129
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:58:39.065728 22396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:58:39.067106 22396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:58:39.082231 22421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:58:39.082257 22422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:58:39.083159 22424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:58:39.085444 22396 server_base.cc:1034] running on GCE node
I20260925 04:58:39.086156 22396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:58:39.087314 22396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:58:39.088537 22396 hybrid_clock.cc:648] HybridClock initialized: now 1790312319088475 us; error 58 us; skew 500 ppm
I20260925 04:58:39.088886 22396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:58:39 dist-test-slave-xzjr krb5kdc[21774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312319, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:58:39.092242 22396 init.cc:377] Logged in from keytab as kudu/127.6.29.129@KRBTEST.COM (short username kudu)
I20260925 04:58:39.093983 22396 webserver.cc:460] Webserver started at http://127.6.29.129:40339/ using document root <none> and password file <none>
I20260925 04:58:39.094722 22396 fs_manager.cc:359] Metadata directory not provided
I20260925 04:58:39.094867 22396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:58:39.100795 22396 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 04:58:39.103890 22431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:58:39.105475 22396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 04:58:39.105667 22396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/wal
uuid: "7071476802644c14854c4de13332b96f"
format_stamp: "Formatted at 2026-09-25 04:58:15 on dist-test-slave-xzjr"
server_key: "6a769dbf6ab00d67342d9c3faac2768b"
server_key_iv: "0797aca69167b9a4f10bff35978bdd08"
server_key_version: "encryptionkey@0"
I20260925 04:58:39.106720 22396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:58:39.120069 22396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:58:39.124637 22396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:58:39.124938 22396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:58:39.126112 22396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:58:39.127641 22396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:58:39.127766 22396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:58:39.127935 22396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:58:39.128028 22396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:58:39.164757 22396 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.129:46341
I20260925 04:58:39.164851 22543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.129:46341 every 8 connection(s)
I20260925 04:58:39.166349 22396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
I20260925 04:58:39.170421 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 22396
I20260925 04:58:39.171162 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.130:35999
--local_ip_for_outbound_sockets=127.6.29.130
--tserver_master_addrs=127.6.29.190:45703
--webserver_port=44231
--webserver_interface=127.6.29.130
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:34331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 04:58:39 dist-test-slave-xzjr krb5kdc[21774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312319, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:58:39.196107 22546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:39.169655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:40039 (local address 127.6.29.190:45703)
0925 04:58:39.170467 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0925 04:58:39.170487 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 04:58:39.171029 (+ 542us) server_negotiation.cc:407] Connection header received
0925 04:58:39.173503 (+ 2474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:58:39.173526 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:58:39.174036 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:58:39.174481 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:58:39.176128 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:39.178317 (+ 2189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:39.180911 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:39.181670 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:39.181834 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:39.186839 (+ 5005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:58:39.186851 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:58:39.186872 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:58:39.186930 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:58:39.190438 (+ 3508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:58:39.191723 (+ 1285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:58:39.191731 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:58:39.191738 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:58:39.191978 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:58:39.192598 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:58:39.192602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:58:39.192605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:58:39.193174 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:58:39.193409 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 04:58:39.194554 (+ 1145us) server_negotiation.cc:299] Negotiation successful
0925 04:58:39.194984 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":312,"threads_started":1}
I20260925 04:58:39.196519 22547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:39.169861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.129:40039)
0925 04:58:39.170815 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:39.170871 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 04:58:39.172831 (+ 1960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:58:39.174642 (+ 1811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:58:39.174650 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:58:39.175196 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:58:39.175889 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:58:39.175927 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:58:39.178531 (+ 2604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:58:39.178536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:58:39.180718 (+ 2182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:58:39.180744 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:58:39.181941 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:58:39.181951 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:58:39.182303 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:39.183149 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:58:39.183186 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:58:39.186435 (+ 3249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:58:39.190672 (+ 4237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:58:39.190686 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:58:39.190709 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:58:39.191528 (+ 819us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:58:39.192109 (+ 581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:58:39.192118 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:58:39.192123 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:58:39.192428 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:58:39.193394 (+ 966us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:58:39.193406 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:58:39.193929 (+ 523us) client_negotiation.cc:769] Sending connection context
0925 04:58:39.194302 (+ 373us) client_negotiation.cc:240] Negotiation successful
0925 04:58:39.194987 (+ 685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":392,"threads_started":1}
I20260925 04:58:39.200057 22544 heartbeater.cc:347] Connected to a master server at 127.6.29.190:45703
I20260925 04:58:39.200420 22544 heartbeater.cc:464] Registering TS with master...
I20260925 04:58:39.201202 22544 heartbeater.cc:511] Master 127.6.29.190:45703 requested a full tablet report, sending...
I20260925 04:58:39.204046 22359 ts_manager.cc:194] Registered new tserver with Master: 7071476802644c14854c4de13332b96f (127.6.29.129:46341)
I20260925 04:58:39.206244 22359 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.129@KRBTEST.COM'} at 127.6.29.129:40039
W20260925 04:58:39.540721 22548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:58:39.541013 22548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:58:39.541107 22548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:58:39.548338 22548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:58:39.548450 22548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:58:39.548563 22548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.130
I20260925 04:58:39.558451 22548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:34331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.130:35999
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.29.130
--webserver_port=44231
--tserver_master_addrs=127.6.29.190:45703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.130
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:58:39.560093 22548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:58:39.561568 22548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:58:39.575634 22553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:58:39.575708 22554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:58:39.575734 22556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:58:39.576653 22548 server_base.cc:1034] running on GCE node
I20260925 04:58:39.577248 22548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:58:39.578176 22548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:58:39.579373 22548 hybrid_clock.cc:648] HybridClock initialized: now 1790312319579335 us; error 33 us; skew 500 ppm
I20260925 04:58:39.579700 22548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:58:39 dist-test-slave-xzjr krb5kdc[21774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312319, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:58:39.582883 22548 init.cc:377] Logged in from keytab as kudu/127.6.29.130@KRBTEST.COM (short username kudu)
I20260925 04:58:39.584440 22548 webserver.cc:460] Webserver started at http://127.6.29.130:44231/ using document root <none> and password file <none>
I20260925 04:58:39.585155 22548 fs_manager.cc:359] Metadata directory not provided
I20260925 04:58:39.585302 22548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:58:39.590893 22548 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260925 04:58:39.593777 22563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:58:39.595160 22548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:58:39.595391 22548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/wal
uuid: "6ac80b17c4a341d09d85f9860bb22c11"
format_stamp: "Formatted at 2026-09-25 04:58:16 on dist-test-slave-xzjr"
server_key: "1b262c63405a21a238c14ccf0bd25c1a"
server_key_iv: "bf8f94fe78ea3a7dfda5ee070e478e63"
server_key_version: "encryptionkey@0"
I20260925 04:58:39.596480 22548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:58:39.616566 22548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:58:39.620982 22548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:58:39.621253 22548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:58:39.622262 22548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:58:39.623615 22548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:58:39.623710 22548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:58:39.623870 22548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:58:39.623960 22548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:58:39.661713 22548 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.130:35999
I20260925 04:58:39.661752 22675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.130:35999 every 8 connection(s)
I20260925 04:58:39.663259 22548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
I20260925 04:58:39.670548 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 22548
I20260925 04:58:39.671339 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.131:33855
--local_ip_for_outbound_sockets=127.6.29.131
--tserver_master_addrs=127.6.29.190:45703
--webserver_port=44155
--webserver_interface=127.6.29.131
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:34331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 04:58:39 dist-test-slave-xzjr krb5kdc[21774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312319, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:58:39.688702 22546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:39.666500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:48565 (local address 127.6.29.190:45703)
0925 04:58:39.666741 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 04:58:39.666748 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:58:39.668119 (+ 1371us) server_negotiation.cc:407] Connection header received
0925 04:58:39.670138 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:58:39.670144 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:58:39.670238 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:58:39.670341 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:58:39.672579 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:39.673304 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:39.675879 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:39.676350 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:39.676463 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:39.681254 (+ 4791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:58:39.681264 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:58:39.681272 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:58:39.681321 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:58:39.685156 (+ 3835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:58:39.686017 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:58:39.686025 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:58:39.686032 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:58:39.686182 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:58:39.686604 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:58:39.686611 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:58:39.686617 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:58:39.687015 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:58:39.687167 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 04:58:39.687967 (+ 800us) server_negotiation.cc:299] Negotiation successful
0925 04:58:39.688236 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 04:58:39.689455 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:39.666979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.130:48565)
0925 04:58:39.667921 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:39.667977 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 04:58:39.669778 (+ 1801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:58:39.670673 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:58:39.670688 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:58:39.671449 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:58:39.672356 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:58:39.672385 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:58:39.673482 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:58:39.673490 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:58:39.675681 (+ 2191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:58:39.675698 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:58:39.676617 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:58:39.676627 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:58:39.676980 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:39.677758 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:58:39.677796 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:58:39.680985 (+ 3189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:58:39.685354 (+ 4369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:58:39.685362 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:58:39.685376 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:58:39.685871 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:58:39.686296 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:58:39.686299 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:58:39.686302 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:58:39.686485 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:58:39.687173 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:58:39.687181 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:58:39.687543 (+ 362us) client_negotiation.cc:769] Sending connection context
0925 04:58:39.687869 (+ 326us) client_negotiation.cc:240] Negotiation successful
0925 04:58:39.688286 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":397,"threads_started":1}
I20260925 04:58:39.691152 22676 heartbeater.cc:347] Connected to a master server at 127.6.29.190:45703
I20260925 04:58:39.691515 22676 heartbeater.cc:464] Registering TS with master...
I20260925 04:58:39.692478 22676 heartbeater.cc:511] Master 127.6.29.190:45703 requested a full tablet report, sending...
I20260925 04:58:39.694123 22359 ts_manager.cc:194] Registered new tserver with Master: 6ac80b17c4a341d09d85f9860bb22c11 (127.6.29.130:35999)
I20260925 04:58:39.695047 22359 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.130@KRBTEST.COM'} at 127.6.29.130:48565
W20260925 04:58:40.019322 22679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:58:40.019626 22679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:58:40.019869 22679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:58:40.027407 22679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:58:40.027529 22679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:58:40.027724 22679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.131
I20260925 04:58:40.038290 22679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:34331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.131:33855
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.29.131
--webserver_port=44155
--tserver_master_addrs=127.6.29.190:45703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.131
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:58:40.039680 22679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:58:40.041085 22679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:58:40.055377 22685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:58:40.055405 22687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:58:40.055361 22684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:58:40.056603 22679 server_base.cc:1034] running on GCE node
I20260925 04:58:40.057233 22679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:58:40.058250 22679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:58:40.059491 22679 hybrid_clock.cc:648] HybridClock initialized: now 1790312320059441 us; error 42 us; skew 500 ppm
I20260925 04:58:40.059923 22679 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:58:40 dist-test-slave-xzjr krb5kdc[21774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312320, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:58:40.063186 22679 init.cc:377] Logged in from keytab as kudu/127.6.29.131@KRBTEST.COM (short username kudu)
I20260925 04:58:40.064880 22679 webserver.cc:460] Webserver started at http://127.6.29.131:44155/ using document root <none> and password file <none>
I20260925 04:58:40.065569 22679 fs_manager.cc:359] Metadata directory not provided
I20260925 04:58:40.065697 22679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:58:40.071238 22679 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260925 04:58:40.074115 22694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:58:40.075664 22679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:58:40.075882 22679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/wal
uuid: "340b5359f50d4360b8e70651f02d5a66"
format_stamp: "Formatted at 2026-09-25 04:58:17 on dist-test-slave-xzjr"
server_key: "9ab89113e5201237eb9f12a1784906fb"
server_key_iv: "8115e489ea68979a8c2d89fa2098ed17"
server_key_version: "encryptionkey@0"
I20260925 04:58:40.076889 22679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:58:40.106508 22679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:58:40.111742 22679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:58:40.112239 22679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:58:40.113410 22679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:58:40.114840 22679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:58:40.114961 22679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:58:40.115092 22679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:58:40.115198 22679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:58:40.153127 22679 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.131:33855
I20260925 04:58:40.153174 22806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.131:33855 every 8 connection(s)
I20260925 04:58:40.154700 22679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
I20260925 04:58:40.161567 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 22679
Sep 25 04:58:40 dist-test-slave-xzjr krb5kdc[21774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312320, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:58:40.178915 22546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:40.157576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:39453 (local address 127.6.29.190:45703)
0925 04:58:40.157931 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0925 04:58:40.157937 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:58:40.158910 (+ 973us) server_negotiation.cc:407] Connection header received
0925 04:58:40.160810 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:58:40.160820 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:58:40.160893 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:58:40.160995 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:58:40.162550 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:40.163547 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:40.166218 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:40.166781 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:40.166911 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:40.171445 (+ 4534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:58:40.171454 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:58:40.171461 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:58:40.171509 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:58:40.175244 (+ 3735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:58:40.176135 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:58:40.176140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:58:40.176142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:58:40.176286 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:58:40.176850 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:58:40.176856 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:58:40.176862 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:58:40.177234 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:58:40.177387 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 04:58:40.178264 (+ 877us) server_negotiation.cc:299] Negotiation successful
0925 04:58:40.178513 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260925 04:58:40.180131 22809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:40.157963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.131:39453)
0925 04:58:40.158746 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:40.158785 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 04:58:40.160547 (+ 1762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:58:40.161199 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:58:40.161208 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:58:40.161689 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:58:40.162294 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:58:40.162338 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:58:40.163723 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:58:40.163727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:58:40.166041 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:58:40.166057 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:58:40.167002 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:58:40.167009 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:58:40.167272 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:40.167998 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:58:40.168021 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:58:40.171149 (+ 3128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:58:40.175393 (+ 4244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:58:40.175406 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:58:40.175429 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:58:40.176001 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:58:40.176442 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:58:40.176448 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:58:40.176453 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:58:40.176715 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:58:40.177383 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:58:40.177396 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:58:40.177849 (+ 453us) client_negotiation.cc:769] Sending connection context
0925 04:58:40.178200 (+ 351us) client_negotiation.cc:240] Negotiation successful
0925 04:58:40.178783 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":326,"threads_started":1}
I20260925 04:58:40.181877 22807 heartbeater.cc:347] Connected to a master server at 127.6.29.190:45703
I20260925 04:58:40.181979 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:40.162569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49198 (local address 127.6.29.190:45703)
0925 04:58:40.163050 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0925 04:58:40.163056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:58:40.163492 (+ 436us) server_negotiation.cc:407] Connection header received
0925 04:58:40.163642 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:58:40.163647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:58:40.163737 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:58:40.163878 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:58:40.165565 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:40.166422 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:40.168593 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:40.169292 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:40.169421 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:40.174604 (+ 5183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:58:40.174610 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:58:40.174613 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:58:40.174655 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:58:40.178597 (+ 3942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:58:40.179512 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:58:40.179518 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:58:40.179521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:58:40.179653 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:58:40.180116 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:58:40.180124 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:58:40.180128 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:58:40.180534 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:58:40.180850 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0925 04:58:40.181321 (+ 471us) server_negotiation.cc:299] Negotiation successful
0925 04:58:40.181560 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":246,"threads_started":1}
I20260925 04:58:40.182253 22807 heartbeater.cc:464] Registering TS with master...
I20260925 04:58:40.183087 22807 heartbeater.cc:511] Master 127.6.29.190:45703 requested a full tablet report, sending...
I20260925 04:58:40.184706 22359 ts_manager.cc:194] Registered new tserver with Master: 340b5359f50d4360b8e70651f02d5a66 (127.6.29.131:33855)
I20260925 04:58:40.185456 22359 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.131@KRBTEST.COM'} at 127.6.29.131:39453
I20260925 04:58:40.197376 6262 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:58:40.204473 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:40.198456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49210 (local address 127.6.29.190:45703)
0925 04:58:40.198631 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 04:58:40.198635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:58:40.199148 (+ 513us) server_negotiation.cc:407] Connection header received
0925 04:58:40.199395 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:58:40.199399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:58:40.199545 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:58:40.199681 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:58:40.200183 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:40.201058 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:40.202649 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:40.203082 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:40.203177 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:40.203665 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:58:40.203954 (+ 289us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 04:58:40.204257 (+ 303us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49210: token expired
Metrics: {"server-negotiator.queue_time_us":64}
W20260925 04:58:40.204622 22810 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49210: token expired
W20260925 04:58:40.205287 22280 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.6.29.190:45703)
I20260925 04:58:40.210716 22544 heartbeater.cc:503] Master 127.6.29.190:45703 was elected leader, sending a full tablet report...
I20260925 04:58:40.219838 22810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:40.205867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49214 (local address 127.6.29.190:45703)
0925 04:58:40.206041 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 04:58:40.206045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:58:40.206060 (+ 15us) server_negotiation.cc:407] Connection header received
0925 04:58:40.206100 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:58:40.206103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:58:40.206159 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:58:40.206235 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:58:40.207732 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:40.208561 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:40.209874 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:40.210378 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:40.210518 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:40.213687 (+ 3169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:58:40.213698 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:58:40.213701 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:58:40.213734 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:58:40.216841 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:58:40.217596 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:58:40.217601 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:58:40.217605 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:58:40.217749 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:58:40.218154 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:58:40.218161 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:58:40.218165 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:58:40.218498 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:58:40.218755 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0925 04:58:40.219179 (+ 424us) server_negotiation.cc:299] Negotiation successful
0925 04:58:40.219426 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 04:58:40.224694 22359 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49214:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:58:40.232580 22359 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:58:40.256244 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:40.245801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48692 (local address 127.6.29.130:35999)
0925 04:58:40.246941 (+ 1140us) server_negotiation.cc:206] Beginning negotiation
0925 04:58:40.246955 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 04:58:40.247464 (+ 509us) server_negotiation.cc:407] Connection header received
0925 04:58:40.247913 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:58:40.247923 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:58:40.248184 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:58:40.248364 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:58:40.249462 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:40.250782 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:40.253529 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:40.255028 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:40.255159 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:40.255390 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0925 04:58:40.255697 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 04:58:40.255883 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":930,"thread_start_us":723,"threads_started":1}
I20260925 04:58:40.256304 22819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:40.245579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:35999 (local address 127.0.0.1:48692)
0925 04:58:40.247309 (+ 1730us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:40.247360 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 04:58:40.247622 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:58:40.248692 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:58:40.248700 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:58:40.248743 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:58:40.249219 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:58:40.249235 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:58:40.250977 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:58:40.250982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:58:40.253344 (+ 2362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:58:40.253356 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:58:40.255263 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:58:40.255269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:58:40.255467 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:40.255505 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 04:58:40.255645 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 04:58:40.255882 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1446,"spinlock_wait_cycles":19840,"thread_start_us":1597,"threads_started":1}
I20260925 04:58:40.257664 22821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:40.247948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.131:33855 (local address 127.0.0.1:54068)
0925 04:58:40.249444 (+ 1496us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:40.249463 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 04:58:40.249570 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:58:40.250763 (+ 1193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:58:40.250767 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:58:40.250793 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:58:40.251271 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:58:40.251283 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:58:40.253220 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:58:40.253225 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:58:40.254817 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:58:40.254829 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:58:40.256756 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:58:40.256762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:58:40.256938 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:40.256973 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 04:58:40.257039 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 04:58:40.257187 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"mutex_wait_us":28,"thread_start_us":945,"threads_started":1}
I20260925 04:58:40.258421 22817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:40.246196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:46341 (local address 127.0.0.1:54560)
0925 04:58:40.247430 (+ 1234us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:40.247448 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:58:40.247621 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:58:40.250054 (+ 2433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:58:40.250059 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:58:40.250084 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:58:40.250436 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:58:40.250448 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:58:40.252575 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:58:40.252581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:58:40.254993 (+ 2412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:58:40.255007 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:58:40.257572 (+ 2565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:58:40.257579 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:58:40.257772 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:40.257814 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 04:58:40.257884 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 04:58:40.258015 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1114,"mutex_wait_us":4,"spinlock_wait_cycles":2304,"thread_start_us":930,"threads_started":1}
I20260925 04:58:40.259434 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:40.246969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54560 (local address 127.6.29.129:46341)
0925 04:58:40.248300 (+ 1331us) server_negotiation.cc:206] Beginning negotiation
0925 04:58:40.249423 (+ 1123us) server_negotiation.cc:399] Waiting for connection header
0925 04:58:40.249458 (+ 35us) server_negotiation.cc:407] Connection header received
0925 04:58:40.249562 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:58:40.249569 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:58:40.249836 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:58:40.250622 (+ 786us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:58:40.250896 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:40.252411 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:40.255162 (+ 2751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:40.257310 (+ 2148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:40.258655 (+ 1345us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:40.258914 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0925 04:58:40.258984 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 04:58:40.259137 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1145,"thread_start_us":1018,"threads_started":1}
I20260925 04:58:40.259752 22822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:40.248539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54068 (local address 127.6.29.131:33855)
0925 04:58:40.250150 (+ 1611us) server_negotiation.cc:206] Beginning negotiation
0925 04:58:40.250159 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:58:40.250210 (+ 51us) server_negotiation.cc:407] Connection header received
0925 04:58:40.250301 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:58:40.250309 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:58:40.250545 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:58:40.250730 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:58:40.251462 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:40.253059 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:40.254986 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:40.256488 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:40.258325 (+ 1837us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:40.258974 (+ 649us) server_negotiation.cc:1035] Waiting for connection context
0925 04:58:40.259054 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 04:58:40.259250 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1437,"thread_start_us":1244,"threads_started":1}
I20260925 04:58:40.260907 22611 tablet_service.cc:1467] Processing CreateTablet for tablet 827663246a9546abbadd24c44b32e1f2 (DEFAULT_TABLE table=test-table [id=633b7b1cbdf14fab8686677a7ee64ebc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:58:40.262519 22611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 827663246a9546abbadd24c44b32e1f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:58:40.263424 22742 tablet_service.cc:1467] Processing CreateTablet for tablet 827663246a9546abbadd24c44b32e1f2 (DEFAULT_TABLE table=test-table [id=633b7b1cbdf14fab8686677a7ee64ebc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:58:40.263343 22479 tablet_service.cc:1467] Processing CreateTablet for tablet 827663246a9546abbadd24c44b32e1f2 (DEFAULT_TABLE table=test-table [id=633b7b1cbdf14fab8686677a7ee64ebc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:58:40.264678 22742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 827663246a9546abbadd24c44b32e1f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:58:40.264847 22479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 827663246a9546abbadd24c44b32e1f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:58:40.277221 22823 tablet_bootstrap.cc:492] T 827663246a9546abbadd24c44b32e1f2 P 340b5359f50d4360b8e70651f02d5a66: Bootstrap starting.
I20260925 04:58:40.279436 22824 tablet_bootstrap.cc:492] T 827663246a9546abbadd24c44b32e1f2 P 6ac80b17c4a341d09d85f9860bb22c11: Bootstrap starting.
I20260925 04:58:40.282299 22825 tablet_bootstrap.cc:492] T 827663246a9546abbadd24c44b32e1f2 P 7071476802644c14854c4de13332b96f: Bootstrap starting.
I20260925 04:58:40.282517 22823 tablet_bootstrap.cc:654] T 827663246a9546abbadd24c44b32e1f2 P 340b5359f50d4360b8e70651f02d5a66: Neither blocks nor log segments found. Creating new log.
I20260925 04:58:40.283725 22824 tablet_bootstrap.cc:654] T 827663246a9546abbadd24c44b32e1f2 P 6ac80b17c4a341d09d85f9860bb22c11: Neither blocks nor log segments found. Creating new log.
I20260925 04:58:40.283715 22823 log.cc:824] T 827663246a9546abbadd24c44b32e1f2 P 340b5359f50d4360b8e70651f02d5a66: Log is configured to *not* fsync() on all Append() calls
I20260925 04:58:40.285050 22824 log.cc:824] T 827663246a9546abbadd24c44b32e1f2 P 6ac80b17c4a341d09d85f9860bb22c11: Log is configured to *not* fsync() on all Append() calls
I20260925 04:58:40.286008 22825 tablet_bootstrap.cc:654] T 827663246a9546abbadd24c44b32e1f2 P 7071476802644c14854c4de13332b96f: Neither blocks nor log segments found. Creating new log.
I20260925 04:58:40.286798 22823 tablet_bootstrap.cc:492] T 827663246a9546abbadd24c44b32e1f2 P 340b5359f50d4360b8e70651f02d5a66: No bootstrap required, opened a new log
I20260925 04:58:40.287065 22823 ts_tablet_manager.cc:1397] T 827663246a9546abbadd24c44b32e1f2 P 340b5359f50d4360b8e70651f02d5a66: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260925 04:58:40.287184 22825 log.cc:824] T 827663246a9546abbadd24c44b32e1f2 P 7071476802644c14854c4de13332b96f: Log is configured to *not* fsync() on all Append() calls
I20260925 04:58:40.288130 22824 tablet_bootstrap.cc:492] T 827663246a9546abbadd24c44b32e1f2 P 6ac80b17c4a341d09d85f9860bb22c11: No bootstrap required, opened a new log
I20260925 04:58:40.288401 22824 ts_tablet_manager.cc:1397] T 827663246a9546abbadd24c44b32e1f2 P 6ac80b17c4a341d09d85f9860bb22c11: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260925 04:58:40.290997 22825 tablet_bootstrap.cc:492] T 827663246a9546abbadd24c44b32e1f2 P 7071476802644c14854c4de13332b96f: No bootstrap required, opened a new log
I20260925 04:58:40.291302 22825 ts_tablet_manager.cc:1397] T 827663246a9546abbadd24c44b32e1f2 P 7071476802644c14854c4de13332b96f: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260925 04:58:40.295046 22823 raft_consensus.cc:348] T 827663246a9546abbadd24c44b32e1f2 P 340b5359f50d4360b8e70651f02d5a66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac80b17c4a341d09d85f9860bb22c11" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35999 } } peers { permanent_uuid: "7071476802644c14854c4de13332b96f" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 46341 } } peers { permanent_uuid: "340b5359f50d4360b8e70651f02d5a66" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 33855 } }
I20260925 04:58:40.295485 22823 raft_consensus.cc:374] T 827663246a9546abbadd24c44b32e1f2 P 340b5359f50d4360b8e70651f02d5a66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:58:40.295617 22823 raft_consensus.cc:729] T 827663246a9546abbadd24c44b32e1f2 P 340b5359f50d4360b8e70651f02d5a66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 340b5359f50d4360b8e70651f02d5a66, State: Initialized, Role: FOLLOWER
I20260925 04:58:40.296280 22823 consensus_queue.cc:260] T 827663246a9546abbadd24c44b32e1f2 P 340b5359f50d4360b8e70651f02d5a66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac80b17c4a341d09d85f9860bb22c11" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35999 } } peers { permanent_uuid: "7071476802644c14854c4de13332b96f" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 46341 } } peers { permanent_uuid: "340b5359f50d4360b8e70651f02d5a66" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 33855 } }
I20260925 04:58:40.297421 22807 heartbeater.cc:503] Master 127.6.29.190:45703 was elected leader, sending a full tablet report...
I20260925 04:58:40.297952 22823 ts_tablet_manager.cc:1428] T 827663246a9546abbadd24c44b32e1f2 P 340b5359f50d4360b8e70651f02d5a66: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260925 04:58:40.299075 22824 raft_consensus.cc:348] T 827663246a9546abbadd24c44b32e1f2 P 6ac80b17c4a341d09d85f9860bb22c11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac80b17c4a341d09d85f9860bb22c11" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35999 } } peers { permanent_uuid: "7071476802644c14854c4de13332b96f" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 46341 } } peers { permanent_uuid: "340b5359f50d4360b8e70651f02d5a66" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 33855 } }
I20260925 04:58:40.299517 22824 raft_consensus.cc:374] T 827663246a9546abbadd24c44b32e1f2 P 6ac80b17c4a341d09d85f9860bb22c11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:58:40.299646 22824 raft_consensus.cc:729] T 827663246a9546abbadd24c44b32e1f2 P 6ac80b17c4a341d09d85f9860bb22c11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ac80b17c4a341d09d85f9860bb22c11, State: Initialized, Role: FOLLOWER
I20260925 04:58:40.300447 22824 consensus_queue.cc:260] T 827663246a9546abbadd24c44b32e1f2 P 6ac80b17c4a341d09d85f9860bb22c11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac80b17c4a341d09d85f9860bb22c11" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35999 } } peers { permanent_uuid: "7071476802644c14854c4de13332b96f" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 46341 } } peers { permanent_uuid: "340b5359f50d4360b8e70651f02d5a66" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 33855 } }
I20260925 04:58:40.302603 22825 raft_consensus.cc:348] T 827663246a9546abbadd24c44b32e1f2 P 7071476802644c14854c4de13332b96f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac80b17c4a341d09d85f9860bb22c11" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35999 } } peers { permanent_uuid: "7071476802644c14854c4de13332b96f" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 46341 } } peers { permanent_uuid: "340b5359f50d4360b8e70651f02d5a66" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 33855 } }
I20260925 04:58:40.302999 22825 raft_consensus.cc:374] T 827663246a9546abbadd24c44b32e1f2 P 7071476802644c14854c4de13332b96f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:58:40.303153 22825 raft_consensus.cc:729] T 827663246a9546abbadd24c44b32e1f2 P 7071476802644c14854c4de13332b96f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7071476802644c14854c4de13332b96f, State: Initialized, Role: FOLLOWER
I20260925 04:58:40.303567 22676 heartbeater.cc:503] Master 127.6.29.190:45703 was elected leader, sending a full tablet report...
I20260925 04:58:40.303871 22825 consensus_queue.cc:260] T 827663246a9546abbadd24c44b32e1f2 P 7071476802644c14854c4de13332b96f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac80b17c4a341d09d85f9860bb22c11" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35999 } } peers { permanent_uuid: "7071476802644c14854c4de13332b96f" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 46341 } } peers { permanent_uuid: "340b5359f50d4360b8e70651f02d5a66" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 33855 } }
I20260925 04:58:40.304301 22824 ts_tablet_manager.cc:1428] T 827663246a9546abbadd24c44b32e1f2 P 6ac80b17c4a341d09d85f9860bb22c11: Time spent starting tablet: real 0.016s user 0.015s sys 0.001s
I20260925 04:58:40.306919 22825 ts_tablet_manager.cc:1428] T 827663246a9546abbadd24c44b32e1f2 P 7071476802644c14854c4de13332b96f: Time spent starting tablet: real 0.016s user 0.010s sys 0.003s
W20260925 04:58:40.406411 22808 tablet.cc:2369] T 827663246a9546abbadd24c44b32e1f2 P 340b5359f50d4360b8e70651f02d5a66: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:58:40.415475 22677 tablet.cc:2369] T 827663246a9546abbadd24c44b32e1f2 P 6ac80b17c4a341d09d85f9860bb22c11: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:58:40.418944 22545 tablet.cc:2369] T 827663246a9546abbadd24c44b32e1f2 P 7071476802644c14854c4de13332b96f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:58:40.553275 22831 raft_consensus.cc:482] T 827663246a9546abbadd24c44b32e1f2 P 7071476802644c14854c4de13332b96f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:58:40.553565 22831 raft_consensus.cc:504] T 827663246a9546abbadd24c44b32e1f2 P 7071476802644c14854c4de13332b96f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac80b17c4a341d09d85f9860bb22c11" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35999 } } peers { permanent_uuid: "7071476802644c14854c4de13332b96f" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 46341 } } peers { permanent_uuid: "340b5359f50d4360b8e70651f02d5a66" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 33855 } }
I20260925 04:58:40.555560 22831 leader_election.cc:288] T 827663246a9546abbadd24c44b32e1f2 P 7071476802644c14854c4de13332b96f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ac80b17c4a341d09d85f9860bb22c11 (127.6.29.130:35999), 340b5359f50d4360b8e70651f02d5a66 (127.6.29.131:33855)
I20260925 04:58:40.561991 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:40.555993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:45691 (local address 127.6.29.130:35999)
0925 04:58:40.556267 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0925 04:58:40.556277 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:58:40.556748 (+ 471us) server_negotiation.cc:407] Connection header received
0925 04:58:40.556975 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:58:40.556988 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:58:40.557049 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:58:40.557147 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:58:40.557678 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:40.558393 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:40.560293 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:40.561052 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:40.561243 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:40.561318 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 04:58:40.561594 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 04:58:40.561736 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 04:58:40.562062 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:40.555906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:35999 (local address 127.6.29.129:45691)
0925 04:58:40.556638 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:40.556653 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:58:40.556828 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:58:40.557165 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:58:40.557175 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:58:40.557199 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:58:40.557543 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:58:40.557552 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:58:40.558532 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:58:40.558540 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:58:40.560132 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:58:40.560148 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:58:40.561257 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:58:40.561262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:58:40.561429 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:40.561453 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 04:58:40.561554 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 04:58:40.561703 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":395,"threads_started":1}
I20260925 04:58:40.562440 22822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:40.556221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:37025 (local address 127.6.29.131:33855)
0925 04:58:40.556460 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 04:58:40.556466 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:58:40.557333 (+ 867us) server_negotiation.cc:407] Connection header received
0925 04:58:40.557541 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:58:40.557546 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:58:40.557613 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:58:40.557719 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:58:40.558221 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:40.558937 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:40.560737 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:40.561509 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:40.561704 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:40.561779 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 04:58:40.562054 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 04:58:40.562190 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 04:58:40.562672 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:40.556507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.131:33855 (local address 127.6.29.129:37025)
0925 04:58:40.557219 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:40.557233 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 04:58:40.557399 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:58:40.557733 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:58:40.557737 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:58:40.557759 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:58:40.558064 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:58:40.558072 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:58:40.559081 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:58:40.559086 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:58:40.560582 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:58:40.560594 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:58:40.561740 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:58:40.561746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:58:40.561897 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:40.561920 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 04:58:40.562017 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 04:58:40.562311 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"mutex_wait_us":60,"thread_start_us":317,"threads_started":1}
I20260925 04:58:40.563285 22631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827663246a9546abbadd24c44b32e1f2" candidate_uuid: "7071476802644c14854c4de13332b96f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ac80b17c4a341d09d85f9860bb22c11" is_pre_election: true
I20260925 04:58:40.563868 22631 raft_consensus.cc:2454] T 827663246a9546abbadd24c44b32e1f2 P 6ac80b17c4a341d09d85f9860bb22c11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7071476802644c14854c4de13332b96f in term 0.
I20260925 04:58:40.563861 22762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827663246a9546abbadd24c44b32e1f2" candidate_uuid: "7071476802644c14854c4de13332b96f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "340b5359f50d4360b8e70651f02d5a66" is_pre_election: true
I20260925 04:58:40.564390 22762 raft_consensus.cc:2454] T 827663246a9546abbadd24c44b32e1f2 P 340b5359f50d4360b8e70651f02d5a66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7071476802644c14854c4de13332b96f in term 0.
I20260925 04:58:40.564602 22435 leader_election.cc:302] T 827663246a9546abbadd24c44b32e1f2 P 7071476802644c14854c4de13332b96f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ac80b17c4a341d09d85f9860bb22c11, 7071476802644c14854c4de13332b96f; no voters:
I20260925 04:58:40.565011 22831 raft_consensus.cc:2781] T 827663246a9546abbadd24c44b32e1f2 P 7071476802644c14854c4de13332b96f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:58:40.565166 22831 raft_consensus.cc:482] T 827663246a9546abbadd24c44b32e1f2 P 7071476802644c14854c4de13332b96f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:58:40.565272 22831 raft_consensus.cc:3037] T 827663246a9546abbadd24c44b32e1f2 P 7071476802644c14854c4de13332b96f [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:58:40.567415 22831 raft_consensus.cc:504] T 827663246a9546abbadd24c44b32e1f2 P 7071476802644c14854c4de13332b96f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac80b17c4a341d09d85f9860bb22c11" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35999 } } peers { permanent_uuid: "7071476802644c14854c4de13332b96f" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 46341 } } peers { permanent_uuid: "340b5359f50d4360b8e70651f02d5a66" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 33855 } }
I20260925 04:58:40.568096 22831 leader_election.cc:288] T 827663246a9546abbadd24c44b32e1f2 P 7071476802644c14854c4de13332b96f [CANDIDATE]: Term 1 election: Requested vote from peers 6ac80b17c4a341d09d85f9860bb22c11 (127.6.29.130:35999), 340b5359f50d4360b8e70651f02d5a66 (127.6.29.131:33855)
I20260925 04:58:40.568508 22631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827663246a9546abbadd24c44b32e1f2" candidate_uuid: "7071476802644c14854c4de13332b96f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ac80b17c4a341d09d85f9860bb22c11"
I20260925 04:58:40.568598 22762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "827663246a9546abbadd24c44b32e1f2" candidate_uuid: "7071476802644c14854c4de13332b96f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "340b5359f50d4360b8e70651f02d5a66"
I20260925 04:58:40.568710 22631 raft_consensus.cc:3037] T 827663246a9546abbadd24c44b32e1f2 P 6ac80b17c4a341d09d85f9860bb22c11 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:58:40.568852 22762 raft_consensus.cc:3037] T 827663246a9546abbadd24c44b32e1f2 P 340b5359f50d4360b8e70651f02d5a66 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:58:40.571002 22762 raft_consensus.cc:2454] T 827663246a9546abbadd24c44b32e1f2 P 340b5359f50d4360b8e70651f02d5a66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7071476802644c14854c4de13332b96f in term 1.
I20260925 04:58:40.571015 22631 raft_consensus.cc:2454] T 827663246a9546abbadd24c44b32e1f2 P 6ac80b17c4a341d09d85f9860bb22c11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7071476802644c14854c4de13332b96f in term 1.
I20260925 04:58:40.571461 22435 leader_election.cc:302] T 827663246a9546abbadd24c44b32e1f2 P 7071476802644c14854c4de13332b96f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ac80b17c4a341d09d85f9860bb22c11, 7071476802644c14854c4de13332b96f; no voters:
I20260925 04:58:40.571749 22831 raft_consensus.cc:2781] T 827663246a9546abbadd24c44b32e1f2 P 7071476802644c14854c4de13332b96f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:58:40.572240 22831 raft_consensus.cc:686] T 827663246a9546abbadd24c44b32e1f2 P 7071476802644c14854c4de13332b96f [term 1 LEADER]: Becoming Leader. State: Replica: 7071476802644c14854c4de13332b96f, State: Running, Role: LEADER
I20260925 04:58:40.572754 22831 consensus_queue.cc:237] T 827663246a9546abbadd24c44b32e1f2 P 7071476802644c14854c4de13332b96f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac80b17c4a341d09d85f9860bb22c11" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35999 } } peers { permanent_uuid: "7071476802644c14854c4de13332b96f" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 46341 } } peers { permanent_uuid: "340b5359f50d4360b8e70651f02d5a66" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 33855 } }
I20260925 04:58:40.577430 22358 catalog_manager.cc:5495] T 827663246a9546abbadd24c44b32e1f2 P 7071476802644c14854c4de13332b96f reported cstate change: term changed from 0 to 1, leader changed from <none> to 7071476802644c14854c4de13332b96f (127.6.29.129). New cstate: current_term: 1 leader_uuid: "7071476802644c14854c4de13332b96f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ac80b17c4a341d09d85f9860bb22c11" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7071476802644c14854c4de13332b96f" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 46341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "340b5359f50d4360b8e70651f02d5a66" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 33855 } health_report { overall_health: UNKNOWN } } }
I20260925 04:58:40.629360 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:40.623870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54568 (local address 127.6.29.129:46341)
0925 04:58:40.624094 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 04:58:40.624102 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:58:40.624123 (+ 21us) server_negotiation.cc:407] Connection header received
0925 04:58:40.624229 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:58:40.624233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:58:40.624318 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:58:40.624479 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:58:40.624883 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:40.625638 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:40.627078 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:40.627493 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:40.627622 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:40.628310 (+ 688us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:58:40.628859 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:58:40.629002 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 04:58:40.629063 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 04:58:40.629131 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 04:58:40.638118 22762 raft_consensus.cc:1261] T 827663246a9546abbadd24c44b32e1f2 P 340b5359f50d4360b8e70651f02d5a66 [term 1 FOLLOWER]: Refusing update from remote peer 7071476802644c14854c4de13332b96f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:58:40.638115 22631 raft_consensus.cc:1261] T 827663246a9546abbadd24c44b32e1f2 P 6ac80b17c4a341d09d85f9860bb22c11 [term 1 FOLLOWER]: Refusing update from remote peer 7071476802644c14854c4de13332b96f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:58:40.639122 22834 consensus_queue.cc:1035] T 827663246a9546abbadd24c44b32e1f2 P 7071476802644c14854c4de13332b96f [LEADER]: Connected to new peer: Peer: permanent_uuid: "340b5359f50d4360b8e70651f02d5a66" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 33855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:58:40.639468 22831 consensus_queue.cc:1035] T 827663246a9546abbadd24c44b32e1f2 P 7071476802644c14854c4de13332b96f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ac80b17c4a341d09d85f9860bb22c11" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 35999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:58:40.673240 22359 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:58:40.673558 22359 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49214:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:58:40.677667 22359 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b: Sending DeleteTablet for 3 replicas of tablet 827663246a9546abbadd24c44b32e1f2
I20260925 04:58:40.678910 22479 tablet_service.cc:1514] Processing DeleteTablet for tablet 827663246a9546abbadd24c44b32e1f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:58:40 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:54560
I20260925 04:58:40.678937 22611 tablet_service.cc:1514] Processing DeleteTablet for tablet 827663246a9546abbadd24c44b32e1f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:58:40 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:48692
I20260925 04:58:40.680595 22742 tablet_service.cc:1514] Processing DeleteTablet for tablet 827663246a9546abbadd24c44b32e1f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:58:40 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:54068
I20260925 04:58:40.681016 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 22396
I20260925 04:58:40.697361 22848 tablet_replica.cc:330] T 827663246a9546abbadd24c44b32e1f2 P 340b5359f50d4360b8e70651f02d5a66: stopping tablet replica
I20260925 04:58:40.697883 22848 raft_consensus.cc:2229] T 827663246a9546abbadd24c44b32e1f2 P 340b5359f50d4360b8e70651f02d5a66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:58:40.698488 22848 raft_consensus.cc:2258] T 827663246a9546abbadd24c44b32e1f2 P 340b5359f50d4360b8e70651f02d5a66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:58:40.699262 22847 tablet_replica.cc:330] T 827663246a9546abbadd24c44b32e1f2 P 6ac80b17c4a341d09d85f9860bb22c11: stopping tablet replica
I20260925 04:58:40.699743 22847 raft_consensus.cc:2229] T 827663246a9546abbadd24c44b32e1f2 P 6ac80b17c4a341d09d85f9860bb22c11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:58:40.700534 22847 raft_consensus.cc:2258] T 827663246a9546abbadd24c44b32e1f2 P 6ac80b17c4a341d09d85f9860bb22c11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:58:40.701297 22848 ts_tablet_manager.cc:1905] T 827663246a9546abbadd24c44b32e1f2 P 340b5359f50d4360b8e70651f02d5a66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:58:40.701431 22846 tablet_replica.cc:330] T 827663246a9546abbadd24c44b32e1f2 P 7071476802644c14854c4de13332b96f: stopping tablet replica
I20260925 04:58:40.701826 22846 raft_consensus.cc:2229] T 827663246a9546abbadd24c44b32e1f2 P 7071476802644c14854c4de13332b96f [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:58:40.702515 22846 raft_consensus.cc:2258] T 827663246a9546abbadd24c44b32e1f2 P 7071476802644c14854c4de13332b96f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:58:40.702698 22847 ts_tablet_manager.cc:1905] T 827663246a9546abbadd24c44b32e1f2 P 6ac80b17c4a341d09d85f9860bb22c11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:58:40.708870 22848 ts_tablet_manager.cc:1918] T 827663246a9546abbadd24c44b32e1f2 P 340b5359f50d4360b8e70651f02d5a66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:58:40.709116 22848 log.cc:1196] T 827663246a9546abbadd24c44b32e1f2 P 340b5359f50d4360b8e70651f02d5a66: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/wal/wals/827663246a9546abbadd24c44b32e1f2
I20260925 04:58:40.709772 22848 ts_tablet_manager.cc:1939] T 827663246a9546abbadd24c44b32e1f2 P 340b5359f50d4360b8e70651f02d5a66: Deleting consensus metadata
I20260925 04:58:40.709921 22847 ts_tablet_manager.cc:1918] T 827663246a9546abbadd24c44b32e1f2 P 6ac80b17c4a341d09d85f9860bb22c11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:58:40.710176 22847 log.cc:1196] T 827663246a9546abbadd24c44b32e1f2 P 6ac80b17c4a341d09d85f9860bb22c11: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/wal/wals/827663246a9546abbadd24c44b32e1f2
I20260925 04:58:40.710796 22847 ts_tablet_manager.cc:1939] T 827663246a9546abbadd24c44b32e1f2 P 6ac80b17c4a341d09d85f9860bb22c11: Deleting consensus metadata
I20260925 04:58:40.712287 22344 catalog_manager.cc:4841] TS 340b5359f50d4360b8e70651f02d5a66 (127.6.29.131:33855): tablet 827663246a9546abbadd24c44b32e1f2 (table test-table [id=633b7b1cbdf14fab8686677a7ee64ebc]) successfully deleted
I20260925 04:58:40.712831 22346 catalog_manager.cc:4841] TS 6ac80b17c4a341d09d85f9860bb22c11 (127.6.29.130:35999): tablet 827663246a9546abbadd24c44b32e1f2 (table test-table [id=633b7b1cbdf14fab8686677a7ee64ebc]) successfully deleted
I20260925 04:58:40.713856 22846 ts_tablet_manager.cc:1905] T 827663246a9546abbadd24c44b32e1f2 P 7071476802644c14854c4de13332b96f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:58:40.716080 22820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:40.682342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54578 (local address 127.6.29.129:46341)
0925 04:58:40.682510 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 04:58:40.682516 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:58:40.696693 (+ 14177us) server_negotiation.cc:407] Connection header received
0925 04:58:40.696893 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:58:40.696900 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:58:40.697013 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:58:40.697121 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:58:40.700024 (+ 2903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:40.701605 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:40.703648 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:40.704366 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:40.704465 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:40.708073 (+ 3608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:58:40.708087 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:58:40.708106 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:58:40.708160 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:58:40.712724 (+ 4564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:58:40.713559 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:58:40.713567 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:58:40.713571 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:58:40.713732 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:58:40.714217 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:58:40.714222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:58:40.714225 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:58:40.714707 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:58:40.715040 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0925 04:58:40.715359 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 04:58:40.715634 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 04:58:40.716884 22539 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:58:40.817168 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 22396
I20260925 04:58:40.835683 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 22548
W20260925 04:58:40.835819 22344 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.29.129:46341 (error 108)
I20260925 04:58:40.837802 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:40.837182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:46341 (local address 127.0.0.1:54582)
0925 04:58:40.837547 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:40.837641 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.129:46341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":181,"threads_started":1}
W20260925 04:58:40.838070 22344 catalog_manager.cc:4584] TS 7071476802644c14854c4de13332b96f (127.6.29.129:46341): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 827663246a9546abbadd24c44b32e1f2: Network error: Client connection negotiation failed: client connection to 127.6.29.129:46341: connect: Connection refused (error 111)
I20260925 04:58:40.850757 22818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:40.836540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48704 (local address 127.6.29.130:35999)
0925 04:58:40.836822 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0925 04:58:40.836829 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:58:40.836995 (+ 166us) server_negotiation.cc:407] Connection header received
0925 04:58:40.837779 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:58:40.837784 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:58:40.837852 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:58:40.837962 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:58:40.839702 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:40.840634 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:40.842032 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:40.842454 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:40.842565 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:40.845212 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:58:40.845226 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:58:40.845241 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:58:40.845278 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:58:40.848348 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:58:40.848965 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:58:40.848973 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:58:40.848975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:58:40.849110 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:58:40.849448 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:58:40.849451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:58:40.849453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:58:40.849823 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:58:40.850033 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0925 04:58:40.850312 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 04:58:40.850471 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260925 04:58:40.851373 22671 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:58:40.891873 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:40.891254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:46341 (local address 127.0.0.1:54598)
0925 04:58:40.891536 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:40.891658 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.129:46341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260925 04:58:40.892997 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:40.892570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:46341 (local address 127.0.0.1:54604)
0925 04:58:40.892746 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:40.892833 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.129:46341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 04:58:40.942852 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 22548
I20260925 04:58:40.953961 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:40.953324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:46341 (local address 127.0.0.1:54612)
0925 04:58:40.953603 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:40.953743 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.129:46341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260925 04:58:40.955220 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:40.954720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:46341 (local address 127.0.0.1:54616)
0925 04:58:40.954924 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:40.955029 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.129:46341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 04:58:40.960544 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 22679
I20260925 04:58:40.974774 22822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:40.961375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54082 (local address 127.6.29.131:33855)
0925 04:58:40.961551 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 04:58:40.961555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:58:40.961575 (+ 20us) server_negotiation.cc:407] Connection header received
0925 04:58:40.961712 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:58:40.961719 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:58:40.961794 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:58:40.961882 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:58:40.963379 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:40.964235 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:40.965592 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:58:40.966013 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:58:40.966119 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:58:40.968969 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:58:40.968992 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:58:40.969010 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:58:40.969046 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:58:40.972085 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:58:40.972892 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:58:40.972900 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:58:40.972903 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:58:40.973045 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:58:40.973410 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:58:40.973413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:58:40.973415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:58:40.973777 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:58:40.974010 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0925 04:58:40.974340 (+ 330us) server_negotiation.cc:299] Negotiation successful
0925 04:58:40.974510 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 04:58:40.975451 22802 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:58:41.035878 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:41.035267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:46341 (local address 127.0.0.1:54626)
0925 04:58:41.035506 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:41.035608 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.129:46341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260925 04:58:41.037006 22852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:58:41.036582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:46341 (local address 127.0.0.1:54630)
0925 04:58:41.036781 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 04:58:41.036865 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.129:46341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260925 04:58:41.066890 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 22679
I20260925 04:58:41.083928 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 22326
I20260925 04:58:41.085036 22389 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:58:41.201443 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 22326
I20260925 04:59:01.223924 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:45703
--webserver_interface=127.6.29.190
--webserver_port=46627
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:34331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:45703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:59:01.559406 22856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:59:01.559710 22856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:59:01.559937 22856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:59:01.567248 22856 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 04:59:01.567354 22856 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 04:59:01.567431 22856 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 04:59:01.567510 22856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:59:01.567639 22856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:59:01.567754 22856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:59:01.567854 22856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:59:01.567934 22856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:59:01.578044 22856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:34331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:45703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:45703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=46627
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:59:01.579473 22856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:59:01.580914 22856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:59:01.593859 22861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:01.593950 22864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:01.593890 22862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:59:01.594183 22856 server_base.cc:1034] running on GCE node
I20260925 04:59:01.595311 22856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:59:01.596918 22856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:59:01.598191 22856 hybrid_clock.cc:648] HybridClock initialized: now 1790312341598124 us; error 48 us; skew 500 ppm
I20260925 04:59:01.598555 22856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:59:01 dist-test-slave-xzjr krb5kdc[21774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312341, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:59:01.601987 22856 init.cc:377] Logged in from keytab as kudu/127.6.29.190@KRBTEST.COM (short username kudu)
I20260925 04:59:01.603552 22856 webserver.cc:460] Webserver started at http://127.6.29.190:46627/ using document root <none> and password file <none>
I20260925 04:59:01.604365 22856 fs_manager.cc:359] Metadata directory not provided
I20260925 04:59:01.604506 22856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:59:01.610548 22856 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:59:01.613683 22871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:01.615037 22856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 04:59:01.615237 22856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "c42d92ab26a24f459a6f7dd1b623038b"
format_stamp: "Formatted at 2026-09-25 04:58:15 on dist-test-slave-xzjr"
server_key: "c9b22eb5243941d99b53fa7fd6ed3e91"
server_key_iv: "12720e2e4b5487fbb25b272d43b033e0"
server_key_version: "encryptionkey@0"
I20260925 04:59:01.616343 22856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:59:01.631172 22856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:59:01.636082 22856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:59:01.636397 22856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:59:01.656034 22856 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:45703
I20260925 04:59:01.656059 22922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:45703 every 8 connection(s)
I20260925 04:59:01.657729 22856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:59:01.659085 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 22856
I20260925 04:59:01.660152 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.129:46341
--local_ip_for_outbound_sockets=127.6.29.129
--tserver_master_addrs=127.6.29.190:45703
--webserver_port=40339
--webserver_interface=127.6.29.129
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:34331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:59:01.668457 22923 sys_catalog.cc:263] Verifying existing consensus state
I20260925 04:59:01.672230 22923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b: Bootstrap starting.
I20260925 04:59:01.694932 22923 log.cc:824] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b: Log is configured to *not* fsync() on all Append() calls
I20260925 04:59:01.713208 22923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 04:59:01.713855 22923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b: Bootstrap complete.
I20260925 04:59:01.724917 22923 raft_consensus.cc:348] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42d92ab26a24f459a6f7dd1b623038b" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 45703 } }
I20260925 04:59:01.725989 22923 raft_consensus.cc:729] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c42d92ab26a24f459a6f7dd1b623038b, State: Initialized, Role: FOLLOWER
I20260925 04:59:01.726773 22923 consensus_queue.cc:260] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42d92ab26a24f459a6f7dd1b623038b" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 45703 } }
I20260925 04:59:01.727057 22923 raft_consensus.cc:388] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:59:01.727205 22923 raft_consensus.cc:482] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:59:01.727404 22923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [term 2 FOLLOWER]: Advancing to term 3
I20260925 04:59:01.731855 22923 raft_consensus.cc:504] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42d92ab26a24f459a6f7dd1b623038b" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 45703 } }
I20260925 04:59:01.732429 22923 leader_election.cc:302] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c42d92ab26a24f459a6f7dd1b623038b; no voters:
I20260925 04:59:01.733197 22923 leader_election.cc:288] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 04:59:01.733345 22927 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [term 3 FOLLOWER]: Leader election won for term 3
I20260925 04:59:01.734575 22927 raft_consensus.cc:686] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [term 3 LEADER]: Becoming Leader. State: Replica: c42d92ab26a24f459a6f7dd1b623038b, State: Running, Role: LEADER
I20260925 04:59:01.735406 22927 consensus_queue.cc:237] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42d92ab26a24f459a6f7dd1b623038b" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 45703 } }
I20260925 04:59:01.735797 22923 sys_catalog.cc:564] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:59:01.738375 22928 sys_catalog.cc:455] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c42d92ab26a24f459a6f7dd1b623038b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42d92ab26a24f459a6f7dd1b623038b" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 45703 } } }
I20260925 04:59:01.738780 22928 sys_catalog.cc:458] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [sys.catalog]: This master's current role is: LEADER
I20260925 04:59:01.738464 22929 sys_catalog.cc:455] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c42d92ab26a24f459a6f7dd1b623038b. Latest consensus state: current_term: 3 leader_uuid: "c42d92ab26a24f459a6f7dd1b623038b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c42d92ab26a24f459a6f7dd1b623038b" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 45703 } } }
I20260925 04:59:01.738957 22929 sys_catalog.cc:458] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b [sys.catalog]: This master's current role is: LEADER
I20260925 04:59:01.740018 22933 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:59:01.759020 22933 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:59:01.761125 22933 catalog_manager.cc:1259] Loaded cluster ID: 23f56071d0524962a48d3f716d58e81a
I20260925 04:59:01.761360 22933 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:59:01.766811 22933 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:59:01.769210 22933 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b: Loaded TSK: 0
I20260925 04:59:01.769531 22933 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c42d92ab26a24f459a6f7dd1b623038b: Loaded TSK: 1
F20260925 04:59:01.771323 22933 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f4e270999ad google::LogMessage::Fail() at ??:0
@ 0x7f4e2709c88a google::LogMessage::SendToLog() at ??:0
@ 0x7f4e270994de google::LogMessage::Flush() at ??:0
@ 0x7f4e2709aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4e2adfd7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f4e2adec73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f4e3cac81c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
I20260925 04:59:01.802376 22945 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f4e3cac7783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f4e3cb532a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f4e3cb530c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f4e3cb9d5e9 std::function<>::operator()() at ??:0
@ 0x7f4e3cacbcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f4e3cac9c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f4e3cb51610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f4e3cb51439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f4e29ce5345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4e29ce891c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f4e29ce8799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f4e29cc3eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4e2cf9d6db start_thread at ??:0
@ 0x7f4e25f3361f clone at ??:0
W20260925 04:59:02.049705 22925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:59:02.050083 22925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:59:02.050266 22925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:59:02.057955 22925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:59:02.058144 22925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:59:02.058331 22925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.129
I20260925 04:59:02.068634 22925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:34331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.129:46341
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.29.129
--webserver_port=40339
--tserver_master_addrs=127.6.29.190:45703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.129
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:59:02.070032 22925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:59:02.071384 22925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:59:02.085712 22971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:02.085784 22970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:02.085911 22973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:59:02.086537 22925 server_base.cc:1034] running on GCE node
I20260925 04:59:02.087208 22925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:59:02.088351 22925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:59:02.089617 22925 hybrid_clock.cc:648] HybridClock initialized: now 1790312342089536 us; error 67 us; skew 500 ppm
I20260925 04:59:02.090005 22925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:59:02 dist-test-slave-xzjr krb5kdc[21774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312342, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:59:02.093488 22925 init.cc:377] Logged in from keytab as kudu/127.6.29.129@KRBTEST.COM (short username kudu)
I20260925 04:59:02.095031 22925 webserver.cc:460] Webserver started at http://127.6.29.129:40339/ using document root <none> and password file <none>
I20260925 04:59:02.095814 22925 fs_manager.cc:359] Metadata directory not provided
I20260925 04:59:02.095963 22925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:59:02.101688 22925 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 04:59:02.105353 22980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:02.106920 22925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:59:02.107131 22925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/wal
uuid: "7071476802644c14854c4de13332b96f"
format_stamp: "Formatted at 2026-09-25 04:58:15 on dist-test-slave-xzjr"
server_key: "6a769dbf6ab00d67342d9c3faac2768b"
server_key_iv: "0797aca69167b9a4f10bff35978bdd08"
server_key_version: "encryptionkey@0"
I20260925 04:59:02.108302 22925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:59:02.128078 22925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:59:02.132269 22925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:59:02.132560 22925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:59:02.133736 22925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:59:02.135962 22987 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260925 04:59:02.141499 22987 ts_tablet_manager.cc:1852] T 827663246a9546abbadd24c44b32e1f2 P 7071476802644c14854c4de13332b96f: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260925 04:59:02.142473 22987 ts_tablet_manager.cc:1905] T 827663246a9546abbadd24c44b32e1f2 P 7071476802644c14854c4de13332b96f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:59:02.149714 22987 ts_tablet_manager.cc:1918] T 827663246a9546abbadd24c44b32e1f2 P 7071476802644c14854c4de13332b96f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:59:02.149899 22987 log.cc:1196] T 827663246a9546abbadd24c44b32e1f2 P 7071476802644c14854c4de13332b96f: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/wal/wals/827663246a9546abbadd24c44b32e1f2
I20260925 04:59:02.150300 22987 ts_tablet_manager.cc:1939] T 827663246a9546abbadd24c44b32e1f2 P 7071476802644c14854c4de13332b96f: Deleting consensus metadata
I20260925 04:59:02.150686 22925 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260925 04:59:02.150840 22925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.001s sys 0.000s
I20260925 04:59:02.150988 22925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:59:02.151081 22925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:02.186429 22925 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.129:46341
I20260925 04:59:02.186493 23093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.129:46341 every 8 connection(s)
I20260925 04:59:02.187906 22925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
I20260925 04:59:02.189145 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 22925
I20260925 04:59:02.189992 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.130:35999
--local_ip_for_outbound_sockets=127.6.29.130
--tserver_master_addrs=127.6.29.190:45703
--webserver_port=44231
--webserver_interface=127.6.29.130
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:34331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:59:02.193642 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:02.191333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.129:55605)
0925 04:59:02.192217 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:02.192511 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":387,"threads_started":1}
W20260925 04:59:02.194309 23094 heartbeater.cc:644] Failed to heartbeat to 127.6.29.190:45703 (0 consecutive failures): Network error: Failed to ping master at 127.6.29.190:45703: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
I20260925 04:59:02.195250 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:02.194919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.129:50689)
0925 04:59:02.195072 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:02.195144 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 04:59:02.196230 23096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:02.195890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.129:46353)
0925 04:59:02.196026 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:02.196112 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 04:59:02.196477 23094 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 04:59:02.544720 23097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:59:02.545080 23097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:59:02.545249 23097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:59:02.552637 23097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:59:02.552788 23097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:59:02.552973 23097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.130
I20260925 04:59:02.562791 23097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:34331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.130:35999
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.29.130
--webserver_port=44231
--tserver_master_addrs=127.6.29.190:45703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.130
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:59:02.564173 23097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:59:02.565565 23097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:59:02.579038 23102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:02.579000 23103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:02.579070 23105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:59:02.580117 23097 server_base.cc:1034] running on GCE node
I20260925 04:59:02.580973 23097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:59:02.581895 23097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:59:02.583153 23097 hybrid_clock.cc:648] HybridClock initialized: now 1790312342583111 us; error 36 us; skew 500 ppm
I20260925 04:59:02.583573 23097 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:59:02 dist-test-slave-xzjr krb5kdc[21774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312342, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:59:02.586762 23097 init.cc:377] Logged in from keytab as kudu/127.6.29.130@KRBTEST.COM (short username kudu)
I20260925 04:59:02.588327 23097 webserver.cc:460] Webserver started at http://127.6.29.130:44231/ using document root <none> and password file <none>
I20260925 04:59:02.589020 23097 fs_manager.cc:359] Metadata directory not provided
I20260925 04:59:02.589171 23097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:59:02.594696 23097 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 04:59:02.597535 23112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:02.599076 23097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260925 04:59:02.599288 23097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/wal
uuid: "6ac80b17c4a341d09d85f9860bb22c11"
format_stamp: "Formatted at 2026-09-25 04:58:16 on dist-test-slave-xzjr"
server_key: "1b262c63405a21a238c14ccf0bd25c1a"
server_key_iv: "bf8f94fe78ea3a7dfda5ee070e478e63"
server_key_version: "encryptionkey@0"
I20260925 04:59:02.600353 23097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:59:02.620564 23097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:59:02.624754 23097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:59:02.625022 23097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:59:02.625988 23097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:59:02.627322 23097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:59:02.627442 23097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:02.627576 23097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:59:02.627663 23097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:02.660028 23097 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.130:35999
I20260925 04:59:02.660074 23224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.130:35999 every 8 connection(s)
I20260925 04:59:02.661541 23097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
I20260925 04:59:02.665410 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 23097
I20260925 04:59:02.666196 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.131:33855
--local_ip_for_outbound_sockets=127.6.29.131
--tserver_master_addrs=127.6.29.190:45703
--webserver_port=44155
--webserver_interface=127.6.29.131
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:34331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 04:59:02.667260 23227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:02.664740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.130:56355)
0925 04:59:02.665659 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:02.665988 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":360,"threads_started":1}
W20260925 04:59:02.668071 23225 heartbeater.cc:644] Failed to heartbeat to 127.6.29.190:45703 (0 consecutive failures): Network error: Failed to ping master at 127.6.29.190:45703: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
I20260925 04:59:02.669024 23227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:02.668587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.130:51889)
0925 04:59:02.668780 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:02.668874 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 04:59:02.670053 23227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:02.669603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.130:38515)
0925 04:59:02.669797 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:02.669898 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260925 04:59:02.670341 23225 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 04:59:03.002383 23228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:59:03.002667 23228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:59:03.002761 23228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:59:03.010188 23228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:59:03.010309 23228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:59:03.010423 23228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.131
I20260925 04:59:03.020645 23228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:34331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.131:33855
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.29.131
--webserver_port=44155
--tserver_master_addrs=127.6.29.190:45703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.131
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:59:03.021974 23228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:59:03.023267 23228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:59:03.037236 23236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:03.037210 23234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:03.037258 23233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:59:03.038151 23228 server_base.cc:1034] running on GCE node
I20260925 04:59:03.038818 23228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:59:03.039932 23228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:59:03.041162 23228 hybrid_clock.cc:648] HybridClock initialized: now 1790312343041106 us; error 49 us; skew 500 ppm
I20260925 04:59:03.041522 23228 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:59:03 dist-test-slave-xzjr krb5kdc[21774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312343, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:59:03.044765 23228 init.cc:377] Logged in from keytab as kudu/127.6.29.131@KRBTEST.COM (short username kudu)
I20260925 04:59:03.046219 23228 webserver.cc:460] Webserver started at http://127.6.29.131:44155/ using document root <none> and password file <none>
I20260925 04:59:03.046940 23228 fs_manager.cc:359] Metadata directory not provided
I20260925 04:59:03.047086 23228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:59:03.052642 23228 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 04:59:03.055596 23243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:03.057102 23228 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 04:59:03.057312 23228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/wal
uuid: "340b5359f50d4360b8e70651f02d5a66"
format_stamp: "Formatted at 2026-09-25 04:58:17 on dist-test-slave-xzjr"
server_key: "9ab89113e5201237eb9f12a1784906fb"
server_key_iv: "8115e489ea68979a8c2d89fa2098ed17"
server_key_version: "encryptionkey@0"
I20260925 04:59:03.058344 23228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:59:03.094310 23228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:59:03.099048 23228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:59:03.099342 23228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:59:03.100430 23228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:59:03.101807 23228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:59:03.101933 23228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:03.102059 23228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:59:03.102151 23228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:03.136948 23228 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.131:33855
I20260925 04:59:03.137008 23355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.131:33855 every 8 connection(s)
I20260925 04:59:03.138557 23228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
I20260925 04:59:03.141695 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 23228
I20260925 04:59:03.143939 23358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:03.141611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.131:45893)
0925 04:59:03.142494 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:03.142735 (+ 241us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":340,"threads_started":1}
W20260925 04:59:03.144670 23356 heartbeater.cc:644] Failed to heartbeat to 127.6.29.190:45703 (0 consecutive failures): Network error: Failed to ping master at 127.6.29.190:45703: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
I20260925 04:59:03.145454 23358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:03.145124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.131:52101)
0925 04:59:03.145287 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:03.145354 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 04:59:03.146246 23358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:03.145936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.131:45519)
0925 04:59:03.146071 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:03.146148 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 04:59:03.146464 23356 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 04:59:03.197841 23360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:03.197070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.129:48475)
0925 04:59:03.197533 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:03.197622 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":222,"threads_started":1}
I20260925 04:59:03.671988 23361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:03.670982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.130:53951)
0925 04:59:03.671551 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:03.671651 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":265,"threads_started":1}
I20260925 04:59:04.147893 23362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:04.147027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.131:49627)
0925 04:59:04.147489 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:04.147594 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":213,"threads_started":1}
I20260925 04:59:04.199496 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:04.198780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.129:43851)
0925 04:59:04.199249 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:04.199333 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":236,"threads_started":1}
I20260925 04:59:04.673787 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:04.672991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.130:48345)
0925 04:59:04.673491 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:04.673582 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":256,"threads_started":1}
I20260925 04:59:05.149937 23365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:05.148938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.131:53471)
0925 04:59:05.149516 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:05.149637 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":261,"threads_started":1}
I20260925 04:59:05.201290 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:05.200497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.129:41101)
0925 04:59:05.200959 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:05.201047 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":229,"threads_started":1}
I20260925 04:59:05.675554 23367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:05.674812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.130:55775)
0925 04:59:05.675249 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:05.675333 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":222,"threads_started":1}
I20260925 04:59:06.151724 23368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:06.150968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.131:40677)
0925 04:59:06.151459 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:06.151543 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":285,"threads_started":1}
I20260925 04:59:06.202886 23369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:06.202205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.129:42775)
0925 04:59:06.202625 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:06.202704 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":211,"threads_started":1}
I20260925 04:59:06.677386 23370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:06.676585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.130:57911)
0925 04:59:06.677076 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:06.677176 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":245,"threads_started":1}
I20260925 04:59:07.153530 23371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:07.152803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.131:48695)
0925 04:59:07.153263 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:07.153345 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":239,"threads_started":1}
I20260925 04:59:07.204826 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:07.203978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.129:59797)
0925 04:59:07.204548 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:07.204647 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":311,"threads_started":1}
I20260925 04:59:07.679443 23373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:07.678493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.130:45799)
0925 04:59:07.679052 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:07.679225 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":284,"threads_started":1}
I20260925 04:59:08.155292 23374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:08.154524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.131:39795)
0925 04:59:08.154990 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:08.155076 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":251,"threads_started":1}
I20260925 04:59:08.206459 23375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:08.205683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.129:45907)
0925 04:59:08.206126 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:08.206232 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":237,"threads_started":1}
I20260925 04:59:08.681125 23376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:08.680374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.130:43467)
0925 04:59:08.680846 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:08.680945 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":235,"threads_started":1}
I20260925 04:59:09.157222 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:09.156401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.131:50603)
0925 04:59:09.156925 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:09.157014 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":301,"threads_started":1}
I20260925 04:59:09.208199 23378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:09.207324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.129:56807)
0925 04:59:09.207853 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:09.207938 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":215,"threads_started":1}
I20260925 04:59:09.682938 23379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:09.682129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.130:34271)
0925 04:59:09.682625 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:09.682710 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":226,"threads_started":1}
I20260925 04:59:10.159080 23380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:10.158250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.131:45271)
0925 04:59:10.158777 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:10.158868 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":219,"threads_started":1}
I20260925 04:59:10.209929 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:10.209211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.129:35235)
0925 04:59:10.209683 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:10.209763 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":260,"threads_started":1}
I20260925 04:59:10.684762 23382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:10.684041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.130:37207)
0925 04:59:10.684484 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:10.684569 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":223,"threads_started":1}
I20260925 04:59:11.160835 23383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:11.160118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.131:42131)
0925 04:59:11.160564 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:11.160659 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":234,"threads_started":1}
I20260925 04:59:11.211928 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:11.211013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.129:53601)
0925 04:59:11.211522 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:11.211725 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":255,"threads_started":1}
I20260925 04:59:11.686471 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:11.685673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.130:54065)
0925 04:59:11.686199 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:11.686294 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":260,"threads_started":1}
I20260925 04:59:12.162771 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:12.161845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.131:42175)
0925 04:59:12.162445 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:12.162567 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":341,"threads_started":1}
I20260925 04:59:12.213577 23387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:12.212869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.129:33273)
0925 04:59:12.213320 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:12.213404 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":224,"threads_started":1}
I20260925 04:59:12.688478 23388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:12.687655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.130:39763)
0925 04:59:12.688213 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:12.688307 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":344,"threads_started":1}
I20260925 04:59:13.164527 23389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:13.163740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.131:57875)
0925 04:59:13.164270 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:13.164356 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":249,"threads_started":1}
I20260925 04:59:13.215324 23390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:13.214560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.129:44217)
0925 04:59:13.215051 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:13.215141 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":272,"threads_started":1}
I20260925 04:59:13.690176 23391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:13.689416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.130:52601)
0925 04:59:13.689898 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:13.689997 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":239,"threads_started":1}
I20260925 04:59:14.166152 23392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:14.165438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.131:44055)
0925 04:59:14.165879 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:14.165970 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":230,"threads_started":1}
I20260925 04:59:14.219774 23393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:14.216377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.129:55917)
0925 04:59:14.219415 (+ 3038us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:14.219537 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2886,"thread_start_us":2769,"threads_started":1}
I20260925 04:59:14.692106 23394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:14.691307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.130:43861)
0925 04:59:14.691839 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:14.691942 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":229,"threads_started":1}
I20260925 04:59:15.168082 23395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:15.167234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.131:52359)
0925 04:59:15.167736 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:15.167916 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":256,"threads_started":1}
I20260925 04:59:15.225327 23396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:15.224500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.129:45425)
0925 04:59:15.224976 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:15.225085 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":257,"threads_started":1}
I20260925 04:59:15.694027 23397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:15.693187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.130:41211)
0925 04:59:15.693721 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:15.693826 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":264,"threads_started":1}
I20260925 04:59:16.169925 23398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:16.169091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.131:48329)
0925 04:59:16.169579 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:16.169697 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":257,"threads_started":1}
I20260925 04:59:16.226970 23399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:16.226227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.129:37737)
0925 04:59:16.226682 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:16.226764 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":240,"threads_started":1}
I20260925 04:59:16.695874 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:16.695012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.130:41893)
0925 04:59:16.695507 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:16.695599 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":261,"threads_started":1}
I20260925 04:59:17.171541 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:17.170808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.131:33061)
0925 04:59:17.171278 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:17.171357 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":220,"threads_started":1}
I20260925 04:59:17.228950 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:17.228186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.129:58255)
0925 04:59:17.228660 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:17.228761 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":227,"threads_started":1}
I20260925 04:59:17.697906 23403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:17.697087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.130:54113)
0925 04:59:17.697561 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:17.697714 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":246,"threads_started":1}
I20260925 04:59:18.146505 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 22925
I20260925 04:59:18.169835 23404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:18.147591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44456 (local address 127.6.29.129:46341)
0925 04:59:18.148622 (+ 1031us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:18.148653 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:18.148703 (+ 50us) server_negotiation.cc:407] Connection header received
0925 04:59:18.150561 (+ 1858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:18.150573 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:18.151034 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:18.151494 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:18.153057 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:18.154343 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:18.155650 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:18.156405 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:18.156602 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:18.163324 (+ 6722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:18.163332 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:18.163348 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:59:18.163392 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:18.166429 (+ 3037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:18.167176 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:18.167183 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:18.167190 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:18.167361 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:18.167791 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:18.167796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:18.167799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:18.168348 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:18.168792 (+ 444us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:18.169090 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 04:59:18.169530 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":821,"thread_start_us":651,"threads_started":1}
I20260925 04:59:18.171278 23089 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:59:18.176714 23406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:18.172761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45703 (local address 127.6.29.131:44591)
0925 04:59:18.176369 (+ 3608us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:18.176494 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.190:45703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3447,"thread_start_us":3323,"threads_started":1}
I20260925 04:59:18.263948 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 22925
I20260925 04:59:18.281423 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 23097
I20260925 04:59:18.301029 23408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:18.282396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55962 (local address 127.6.29.130:35999)
0925 04:59:18.282973 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:18.282988 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:18.283028 (+ 40us) server_negotiation.cc:407] Connection header received
0925 04:59:18.284770 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:18.284787 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:18.285249 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:18.285651 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:18.287103 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:18.288429 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:18.289827 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:18.290569 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:18.290780 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:18.293655 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:18.293664 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:18.293681 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:59:18.293722 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:18.297200 (+ 3478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:18.298035 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:18.298043 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:18.298116 (+ 73us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:18.298322 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:18.298742 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:18.298748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:18.298752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:18.299338 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:18.299657 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:18.300256 (+ 599us) server_negotiation.cc:299] Negotiation successful
0925 04:59:18.300729 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":248,"threads_started":1}
I20260925 04:59:18.302629 23220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:59:18.387496 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 23097
I20260925 04:59:18.406716 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 23228
I20260925 04:59:18.425501 23410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:18.407899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57974 (local address 127.6.29.131:33855)
0925 04:59:18.408523 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:18.408537 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:18.408581 (+ 44us) server_negotiation.cc:407] Connection header received
0925 04:59:18.410230 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:18.410241 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:18.410691 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:18.411057 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:18.412776 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:18.414025 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:18.415310 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:18.416124 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:18.416332 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:18.418943 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:18.418951 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:18.418966 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:59:18.419003 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:18.421986 (+ 2983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:18.422765 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:18.422774 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:18.422789 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:18.422961 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:18.423347 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:18.423353 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:18.423356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:18.423981 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:18.424297 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:18.424756 (+ 459us) server_negotiation.cc:299] Negotiation successful
0925 04:59:18.425192 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":286,"threads_started":1}
I20260925 04:59:18.427007 23351 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:59:18.510375 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 23228
2026-09-25T04:59:18Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64960 ms)
[----------] 30 tests from SecurityITest (304805 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:59:18 dist-test-slave-xzjr krb5kdc[23414](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:59:18 dist-test-slave-xzjr krb5kdc[23414](info): set up 2 sockets
Sep 25 04:59:18 dist-test-slave-xzjr krb5kdc[23414](info): commencing operation
krb5kdc: starting...
W20260925 04:59:19.676496 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.008s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 04:59:19 dist-test-slave-xzjr krb5kdc[23414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:59:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:59:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.29.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
I20260925 04:59:19.944646 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:43841
--webserver_interface=127.6.29.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:43001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:43841
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:59:20.290392 23427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:59:20.290787 23427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:59:20.290956 23427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:59:20.298426 23427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:59:20.298569 23427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:59:20.298731 23427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:59:20.298848 23427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:59:20.298923 23427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:59:20.309176 23427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:43001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:43841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:43841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:59:20.310617 23427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:59:20.312021 23427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:59:20.324589 23433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:20.324640 23435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:20.324692 23432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:59:20.325058 23427 server_base.cc:1034] running on GCE node
I20260925 04:59:20.326143 23427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:59:20.327728 23427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:59:20.329020 23427 hybrid_clock.cc:648] HybridClock initialized: now 1790312360328978 us; error 36 us; skew 500 ppm
I20260925 04:59:20.329402 23427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:59:20 dist-test-slave-xzjr krb5kdc[23414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312360, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:59:20.332669 23427 init.cc:377] Logged in from keytab as kudu/127.6.29.190@KRBTEST.COM (short username kudu)
I20260925 04:59:20.334438 23427 webserver.cc:460] Webserver started at http://127.6.29.190:41983/ using document root <none> and password file <none>
I20260925 04:59:20.335184 23427 fs_manager.cc:359] Metadata directory not provided
I20260925 04:59:20.335333 23427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:59:20.335665 23427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:59:20.338923 23427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "0553d55d329d47d38aa10e9dbe3b07a0"
format_stamp: "Formatted at 2026-09-25 04:59:20 on dist-test-slave-xzjr"
server_key: "5fad51172deca8e0136ba0a12cf37da4"
server_key_iv: "99b9b89384e791b09b53ce7ae5395273"
server_key_version: "encryptionkey@0"
I20260925 04:59:20.339721 23427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "0553d55d329d47d38aa10e9dbe3b07a0"
format_stamp: "Formatted at 2026-09-25 04:59:20 on dist-test-slave-xzjr"
server_key: "5fad51172deca8e0136ba0a12cf37da4"
server_key_iv: "99b9b89384e791b09b53ce7ae5395273"
server_key_version: "encryptionkey@0"
I20260925 04:59:20.345239 23427 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260925 04:59:20.348882 23442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:20.350417 23427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 04:59:20.350608 23427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "0553d55d329d47d38aa10e9dbe3b07a0"
format_stamp: "Formatted at 2026-09-25 04:59:20 on dist-test-slave-xzjr"
server_key: "5fad51172deca8e0136ba0a12cf37da4"
server_key_iv: "99b9b89384e791b09b53ce7ae5395273"
server_key_version: "encryptionkey@0"
I20260925 04:59:20.350840 23427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:59:20.365549 23427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:59:20.370090 23427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:59:20.370395 23427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:59:20.389731 23427 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:43841
I20260925 04:59:20.389772 23493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:43841 every 8 connection(s)
I20260925 04:59:20.391175 23427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:59:20.394941 23494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:59:20.400503 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 23427
I20260925 04:59:20.400735 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 04:59:20.401396 6262 external_mini_cluster.cc:1428] Setting key 75877b3d07c682ca39418a8b06d9578e
I20260925 04:59:20.406760 23494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0553d55d329d47d38aa10e9dbe3b07a0: Bootstrap starting.
I20260925 04:59:20.409927 23494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0553d55d329d47d38aa10e9dbe3b07a0: Neither blocks nor log segments found. Creating new log.
I20260925 04:59:20.410986 23494 log.cc:824] T 00000000000000000000000000000000 P 0553d55d329d47d38aa10e9dbe3b07a0: Log is configured to *not* fsync() on all Append() calls
I20260925 04:59:20.414268 23494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0553d55d329d47d38aa10e9dbe3b07a0: No bootstrap required, opened a new log
Sep 25 04:59:20 dist-test-slave-xzjr krb5kdc[23414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:59:20.424352 23494 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0553d55d329d47d38aa10e9dbe3b07a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0553d55d329d47d38aa10e9dbe3b07a0" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43841 } }
I20260925 04:59:20.424896 23494 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0553d55d329d47d38aa10e9dbe3b07a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:59:20.425014 23494 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0553d55d329d47d38aa10e9dbe3b07a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0553d55d329d47d38aa10e9dbe3b07a0, State: Initialized, Role: FOLLOWER
I20260925 04:59:20.425704 23494 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0553d55d329d47d38aa10e9dbe3b07a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0553d55d329d47d38aa10e9dbe3b07a0" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43841 } }
I20260925 04:59:20.426000 23494 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0553d55d329d47d38aa10e9dbe3b07a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:59:20.426144 23494 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0553d55d329d47d38aa10e9dbe3b07a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:59:20.426333 23494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0553d55d329d47d38aa10e9dbe3b07a0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:59:20.426672 23497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:20.403400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36372 (local address 127.6.29.190:43841)
0925 04:59:20.404402 (+ 1002us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:20.404421 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:20.404752 (+ 331us) server_negotiation.cc:407] Connection header received
0925 04:59:20.406671 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:20.406696 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:20.407140 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:20.407535 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:20.409061 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:20.410421 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:20.411975 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:20.413004 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:20.413207 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:20.417404 (+ 4197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:20.417415 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:20.417441 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:59:20.417492 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:20.421696 (+ 4204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:20.422482 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:20.422491 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:20.422500 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:20.422720 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:20.423104 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:20.423108 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:20.423112 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:20.423890 (+ 778us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:20.424211 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:20.424624 (+ 413us) server_negotiation.cc:299] Negotiation successful
0925 04:59:20.425194 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":470,"threads_started":1}
I20260925 04:59:20.429411 23494 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0553d55d329d47d38aa10e9dbe3b07a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0553d55d329d47d38aa10e9dbe3b07a0" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43841 } }
I20260925 04:59:20.430045 23494 leader_election.cc:302] T 00000000000000000000000000000000 P 0553d55d329d47d38aa10e9dbe3b07a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0553d55d329d47d38aa10e9dbe3b07a0; no voters:
I20260925 04:59:20.430677 23494 leader_election.cc:288] T 00000000000000000000000000000000 P 0553d55d329d47d38aa10e9dbe3b07a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:59:20.430835 23499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0553d55d329d47d38aa10e9dbe3b07a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:59:20.431586 23499 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0553d55d329d47d38aa10e9dbe3b07a0 [term 1 LEADER]: Becoming Leader. State: Replica: 0553d55d329d47d38aa10e9dbe3b07a0, State: Running, Role: LEADER
I20260925 04:59:20.432132 23499 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0553d55d329d47d38aa10e9dbe3b07a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0553d55d329d47d38aa10e9dbe3b07a0" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43841 } }
I20260925 04:59:20.432878 23494 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0553d55d329d47d38aa10e9dbe3b07a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:59:20.435221 23501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0553d55d329d47d38aa10e9dbe3b07a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0553d55d329d47d38aa10e9dbe3b07a0. Latest consensus state: current_term: 1 leader_uuid: "0553d55d329d47d38aa10e9dbe3b07a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0553d55d329d47d38aa10e9dbe3b07a0" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43841 } } }
I20260925 04:59:20.435596 23501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0553d55d329d47d38aa10e9dbe3b07a0 [sys.catalog]: This master's current role is: LEADER
I20260925 04:59:20.436060 23500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0553d55d329d47d38aa10e9dbe3b07a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0553d55d329d47d38aa10e9dbe3b07a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0553d55d329d47d38aa10e9dbe3b07a0" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43841 } } }
I20260925 04:59:20.437136 23500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0553d55d329d47d38aa10e9dbe3b07a0 [sys.catalog]: This master's current role is: LEADER
I20260925 04:59:20.438536 23507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:59:20.444657 23507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:59:20.454231 23507 catalog_manager.cc:1347] Generated new cluster ID: 3755c366b2234cd6b8460a0d48ed6b36
I20260925 04:59:20.454367 23507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:59:20.485844 23507 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:59:20.487319 23507 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:59:20.493746 23507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0553d55d329d47d38aa10e9dbe3b07a0: Generated new TSK 0
I20260925 04:59:20.494658 23507 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.29.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab.
Entry for principal kudu/127.6.29.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab.
I20260925 04:59:20.564474 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.129:0
--local_ip_for_outbound_sockets=127.6.29.129
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:43841
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:43001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:59:20.907622 23520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:59:20.907999 23520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:59:20.908104 23520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:59:20.916090 23520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:59:20.916225 23520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:59:20.916342 23520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.129
I20260925 04:59:20.926630 23520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:43001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:43841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.129
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:59:20.928018 23520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:59:20.929448 23520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:59:20.943447 23525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:20.943414 23526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:20.943835 23528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:59:20.944370 23520 server_base.cc:1034] running on GCE node
I20260925 04:59:20.945115 23520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:59:20.946162 23520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:59:20.947439 23520 hybrid_clock.cc:648] HybridClock initialized: now 1790312360947382 us; error 45 us; skew 500 ppm
I20260925 04:59:20.947755 23520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:59:20 dist-test-slave-xzjr krb5kdc[23414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312360, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:59:20.951314 23520 init.cc:377] Logged in from keytab as kudu/127.6.29.129@KRBTEST.COM (short username kudu)
I20260925 04:59:20.953045 23520 webserver.cc:460] Webserver started at http://127.6.29.129:40721/ using document root <none> and password file <none>
I20260925 04:59:20.953815 23520 fs_manager.cc:359] Metadata directory not provided
I20260925 04:59:20.953950 23520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:59:20.954305 23520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:59:20.957526 23520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-0/data/instance:
uuid: "279bee6bb6ba4acea53a23125b276eb5"
format_stamp: "Formatted at 2026-09-25 04:59:20 on dist-test-slave-xzjr"
server_key: "9a6e5186bc4822ea4f97b6d97a27cd23"
server_key_iv: "75d398594a305fe2e012082b31ee376d"
server_key_version: "encryptionkey@0"
I20260925 04:59:20.958372 23520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance:
uuid: "279bee6bb6ba4acea53a23125b276eb5"
format_stamp: "Formatted at 2026-09-25 04:59:20 on dist-test-slave-xzjr"
server_key: "9a6e5186bc4822ea4f97b6d97a27cd23"
server_key_iv: "75d398594a305fe2e012082b31ee376d"
server_key_version: "encryptionkey@0"
I20260925 04:59:20.964139 23520 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 04:59:20.968133 23535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:20.969686 23520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:59:20.969877 23520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-0/wal
uuid: "279bee6bb6ba4acea53a23125b276eb5"
format_stamp: "Formatted at 2026-09-25 04:59:20 on dist-test-slave-xzjr"
server_key: "9a6e5186bc4822ea4f97b6d97a27cd23"
server_key_iv: "75d398594a305fe2e012082b31ee376d"
server_key_version: "encryptionkey@0"
I20260925 04:59:20.970094 23520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:59:20.991416 23520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:59:20.996280 23520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:59:20.996579 23520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:59:20.997545 23520 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:59:20.998991 23520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:59:20.999063 23520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:20.999236 23520 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:59:20.999298 23520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:21.034142 23520 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.129:36181
I20260925 04:59:21.034219 23647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.129:36181 every 8 connection(s)
I20260925 04:59:21.035629 23520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
I20260925 04:59:21.040135 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 23520
I20260925 04:59:21.040395 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance
I20260925 04:59:21.041054 6262 external_mini_cluster.cc:1428] Setting key b0447bac966208c065bd9cf3500de709
Sep 25 04:59:21 dist-test-slave-xzjr krb5kdc[23414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312360, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:59:21.059731 23650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:21.038297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:52913 (local address 127.6.29.190:43841)
0925 04:59:21.038799 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:21.038804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:21.040031 (+ 1227us) server_negotiation.cc:407] Connection header received
0925 04:59:21.042521 (+ 2490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:21.042525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:21.042593 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:21.042684 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:21.044669 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:21.045903 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:21.047709 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:21.048511 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:21.048668 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:21.053334 (+ 4666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:21.053339 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:21.053343 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:59:21.053381 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:21.056237 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:21.057252 (+ 1015us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:21.057256 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:21.057259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:21.057382 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:21.057890 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:21.057894 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:21.057896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:21.058194 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:21.058307 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:21.059282 (+ 975us) server_negotiation.cc:299] Negotiation successful
0925 04:59:21.059457 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":292,"threads_started":1}
I20260925 04:59:21.061363 23651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:21.038941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43841 (local address 127.6.29.129:52913)
0925 04:59:21.039840 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:21.039886 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:21.042164 (+ 2278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:21.043072 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:21.043085 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:21.043749 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:59:21.044475 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:21.044496 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:21.046097 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:21.046101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:21.047543 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:21.047557 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:21.048808 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:21.048814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:21.049070 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:21.049853 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:59:21.049882 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:59:21.053022 (+ 3140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:59:21.056384 (+ 3362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:59:21.056397 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:59:21.056420 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:59:21.057090 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:59:21.057490 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:59:21.057497 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:59:21.057503 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:59:21.057764 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:59:21.058333 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:59:21.058344 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:59:21.058847 (+ 503us) client_negotiation.cc:769] Sending connection context
0925 04:59:21.059175 (+ 328us) client_negotiation.cc:240] Negotiation successful
0925 04:59:21.059844 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":325,"threads_started":1}
I20260925 04:59:21.063149 23648 heartbeater.cc:347] Connected to a master server at 127.6.29.190:43841
I20260925 04:59:21.063501 23648 heartbeater.cc:464] Registering TS with master...
I20260925 04:59:21.064352 23648 heartbeater.cc:511] Master 127.6.29.190:43841 requested a full tablet report, sending...
I20260925 04:59:21.066740 23459 ts_manager.cc:194] Registered new tserver with Master: 279bee6bb6ba4acea53a23125b276eb5 (127.6.29.129:36181)
WARNING: no policy specified for kudu/127.6.29.130@KRBTEST.COM; defaulting to no policy
I20260925 04:59:21.068492 23459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.129@KRBTEST.COM'} at 127.6.29.129:52913
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab.
Entry for principal kudu/127.6.29.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab.
I20260925 04:59:21.102938 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.130:0
--local_ip_for_outbound_sockets=127.6.29.130
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:43841
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:43001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:59:21.442555 23654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:59:21.442888 23654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:59:21.443056 23654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:59:21.450349 23654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:59:21.450498 23654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:59:21.450697 23654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.130
I20260925 04:59:21.460917 23654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:43001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:43841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.130
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:59:21.462297 23654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:59:21.463678 23654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:59:21.476904 23660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:21.476847 23659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:21.476938 23662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:59:21.477713 23654 server_base.cc:1034] running on GCE node
I20260925 04:59:21.478374 23654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:59:21.479314 23654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:59:21.480533 23654 hybrid_clock.cc:648] HybridClock initialized: now 1790312361480491 us; error 37 us; skew 500 ppm
I20260925 04:59:21.480902 23654 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:59:21 dist-test-slave-xzjr krb5kdc[23414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312361, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:59:21.484270 23654 init.cc:377] Logged in from keytab as kudu/127.6.29.130@KRBTEST.COM (short username kudu)
I20260925 04:59:21.485885 23654 webserver.cc:460] Webserver started at http://127.6.29.130:37129/ using document root <none> and password file <none>
I20260925 04:59:21.486608 23654 fs_manager.cc:359] Metadata directory not provided
I20260925 04:59:21.486756 23654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:59:21.487074 23654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:59:21.490096 23654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-1/data/instance:
uuid: "c6cf5c105516413d8866ea1f6f5b9219"
format_stamp: "Formatted at 2026-09-25 04:59:21 on dist-test-slave-xzjr"
server_key: "4007185f27553cb76ccf604690481c43"
server_key_iv: "063850ca58e0903f17e4076d6c5b3f48"
server_key_version: "encryptionkey@0"
I20260925 04:59:21.490888 23654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance:
uuid: "c6cf5c105516413d8866ea1f6f5b9219"
format_stamp: "Formatted at 2026-09-25 04:59:21 on dist-test-slave-xzjr"
server_key: "4007185f27553cb76ccf604690481c43"
server_key_iv: "063850ca58e0903f17e4076d6c5b3f48"
server_key_version: "encryptionkey@0"
I20260925 04:59:21.496387 23654 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 04:59:21.499984 23669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:21.501394 23654 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:59:21.501588 23654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-1/wal
uuid: "c6cf5c105516413d8866ea1f6f5b9219"
format_stamp: "Formatted at 2026-09-25 04:59:21 on dist-test-slave-xzjr"
server_key: "4007185f27553cb76ccf604690481c43"
server_key_iv: "063850ca58e0903f17e4076d6c5b3f48"
server_key_version: "encryptionkey@0"
I20260925 04:59:21.501801 23654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:59:21.524578 23654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:59:21.529404 23654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:59:21.529728 23654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:59:21.530797 23654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:59:21.532274 23654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:59:21.532384 23654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:21.532555 23654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:59:21.532624 23654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:21.568354 23654 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.130:40727
I20260925 04:59:21.568457 23781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.130:40727 every 8 connection(s)
I20260925 04:59:21.569758 23654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
I20260925 04:59:21.578855 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 23654
I20260925 04:59:21.579034 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance
I20260925 04:59:21.579480 6262 external_mini_cluster.cc:1428] Setting key 6a2d32750d7f169d46e54a6cba623669
Sep 25 04:59:21 dist-test-slave-xzjr krb5kdc[23414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312361, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:59:21.592731 23784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:21.572562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:51843 (local address 127.6.29.190:43841)
0925 04:59:21.573043 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:21.573056 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:21.574201 (+ 1145us) server_negotiation.cc:407] Connection header received
0925 04:59:21.576227 (+ 2026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:21.576232 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:21.576304 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:21.576414 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:21.577934 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:21.578964 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:21.580862 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:21.581368 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:21.581509 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:21.585394 (+ 3885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:21.585403 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:21.585411 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:59:21.585464 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:21.589264 (+ 3800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:21.590126 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:21.590134 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:21.590141 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:21.590325 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:21.590896 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:21.590899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:21.590901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:21.591184 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:21.591322 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:21.592261 (+ 939us) server_negotiation.cc:299] Negotiation successful
0925 04:59:21.592457 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":236,"threads_started":1}
I20260925 04:59:21.594324 23785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:21.573132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43841 (local address 127.6.29.130:51843)
0925 04:59:21.574006 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:21.574062 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:21.575920 (+ 1858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:21.576630 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:21.576639 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:21.577169 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:59:21.577739 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:21.577756 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:21.579149 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:21.579157 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:21.580712 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:21.580724 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:21.581617 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:21.581625 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:21.581859 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:21.582602 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:59:21.582630 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:59:21.585117 (+ 2487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:59:21.589445 (+ 4328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:59:21.589453 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:59:21.589467 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:59:21.589945 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:59:21.590465 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:59:21.590471 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:59:21.590477 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:59:21.590775 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:59:21.591323 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:59:21.591332 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:59:21.591862 (+ 530us) client_negotiation.cc:769] Sending connection context
0925 04:59:21.592175 (+ 313us) client_negotiation.cc:240] Negotiation successful
0925 04:59:21.592799 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":380,"threads_started":1}
I20260925 04:59:21.595920 23782 heartbeater.cc:347] Connected to a master server at 127.6.29.190:43841
I20260925 04:59:21.596256 23782 heartbeater.cc:464] Registering TS with master...
I20260925 04:59:21.597016 23782 heartbeater.cc:511] Master 127.6.29.190:43841 requested a full tablet report, sending...
I20260925 04:59:21.598546 23459 ts_manager.cc:194] Registered new tserver with Master: c6cf5c105516413d8866ea1f6f5b9219 (127.6.29.130:40727)
I20260925 04:59:21.599388 23459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.130@KRBTEST.COM'} at 127.6.29.130:51843
WARNING: no policy specified for kudu/127.6.29.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab.
Entry for principal kudu/127.6.29.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab.
I20260925 04:59:21.639346 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.131:0
--local_ip_for_outbound_sockets=127.6.29.131
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:43841
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:43001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:59:21.970821 23788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:59:21.971117 23788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:59:21.971212 23788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:59:21.978799 23788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:59:21.978924 23788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:59:21.979039 23788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.131
I20260925 04:59:21.989418 23788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:43001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:43841
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.131
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:59:21.990726 23788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:59:21.992156 23788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:59:22.006209 23793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:22.006341 23794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:22.006163 23796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:59:22.007292 23788 server_base.cc:1034] running on GCE node
I20260925 04:59:22.008005 23788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:59:22.009023 23788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:59:22.010265 23788 hybrid_clock.cc:648] HybridClock initialized: now 1790312362010188 us; error 38 us; skew 500 ppm
I20260925 04:59:22.010613 23788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:59:22 dist-test-slave-xzjr krb5kdc[23414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312362, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:59:22.013931 23788 init.cc:377] Logged in from keytab as kudu/127.6.29.131@KRBTEST.COM (short username kudu)
I20260925 04:59:22.015419 23788 webserver.cc:460] Webserver started at http://127.6.29.131:44677/ using document root <none> and password file <none>
I20260925 04:59:22.016163 23788 fs_manager.cc:359] Metadata directory not provided
I20260925 04:59:22.016278 23788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:59:22.016530 23788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:59:22.019533 23788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-2/data/instance:
uuid: "f49a44c0fe7b4f168c25cbc628ef4afe"
format_stamp: "Formatted at 2026-09-25 04:59:22 on dist-test-slave-xzjr"
server_key: "6ddf1ed359405c7a136727f648b3d93c"
server_key_iv: "5d31ff90ada890086aea1fb375a40b47"
server_key_version: "encryptionkey@0"
I20260925 04:59:22.020427 23788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance:
uuid: "f49a44c0fe7b4f168c25cbc628ef4afe"
format_stamp: "Formatted at 2026-09-25 04:59:22 on dist-test-slave-xzjr"
server_key: "6ddf1ed359405c7a136727f648b3d93c"
server_key_iv: "5d31ff90ada890086aea1fb375a40b47"
server_key_version: "encryptionkey@0"
I20260925 04:59:22.025856 23788 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 04:59:22.029723 23803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:22.031293 23788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 04:59:22.031514 23788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-2/wal
uuid: "f49a44c0fe7b4f168c25cbc628ef4afe"
format_stamp: "Formatted at 2026-09-25 04:59:22 on dist-test-slave-xzjr"
server_key: "6ddf1ed359405c7a136727f648b3d93c"
server_key_iv: "5d31ff90ada890086aea1fb375a40b47"
server_key_version: "encryptionkey@0"
I20260925 04:59:22.031713 23788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:59:22.063226 23788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:59:22.068022 23788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:59:22.068352 23788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:59:22.069417 23788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:59:22.070835 23788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:59:22.070915 23788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:22.071050 23788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:59:22.071110 23788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:22.072397 23648 heartbeater.cc:503] Master 127.6.29.190:43841 was elected leader, sending a full tablet report...
I20260925 04:59:22.107368 23788 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.131:36869
I20260925 04:59:22.107462 23915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.131:36869 every 8 connection(s)
I20260925 04:59:22.108811 23788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
I20260925 04:59:22.114735 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 23788
I20260925 04:59:22.114934 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance
I20260925 04:59:22.115458 6262 external_mini_cluster.cc:1428] Setting key 47f534f9736a7650394d0ddc6299f316
Sep 25 04:59:22 dist-test-slave-xzjr krb5kdc[23414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312362, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:59:22.133044 23918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:22.111687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:36985 (local address 127.6.29.190:43841)
0925 04:59:22.112305 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:22.112311 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:22.113219 (+ 908us) server_negotiation.cc:407] Connection header received
0925 04:59:22.115489 (+ 2270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:22.115495 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:22.115603 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:22.115749 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:22.118015 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:22.119088 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:22.120932 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:22.121371 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:22.121477 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:22.125204 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:22.125210 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:22.125213 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:59:22.125249 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:22.128928 (+ 3679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:22.129944 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:22.129951 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:22.129954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:22.130123 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:22.130809 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:22.130814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:22.130817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:22.131191 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:22.131374 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:22.132426 (+ 1052us) server_negotiation.cc:299] Negotiation successful
0925 04:59:22.132686 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":356,"threads_started":1}
I20260925 04:59:22.134094 23919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:22.112246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43841 (local address 127.6.29.131:36985)
0925 04:59:22.113010 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:22.113059 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:22.115166 (+ 2107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:22.116106 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:22.116129 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:22.116968 (+ 839us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:59:22.117812 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:22.117838 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:22.119284 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:22.119288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:22.120782 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:22.120796 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:22.121662 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:22.121674 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:22.121999 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:22.122686 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:59:22.122709 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:59:22.124956 (+ 2247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:59:22.129139 (+ 4183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:59:22.129147 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:59:22.129163 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:59:22.129771 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:59:22.130272 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:59:22.130276 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:59:22.130279 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:59:22.130659 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:59:22.131363 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:59:22.131370 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:59:22.131919 (+ 549us) client_negotiation.cc:769] Sending connection context
0925 04:59:22.132300 (+ 381us) client_negotiation.cc:240] Negotiation successful
0925 04:59:22.132765 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":308,"threads_started":1}
I20260925 04:59:22.135543 23916 heartbeater.cc:347] Connected to a master server at 127.6.29.190:43841
I20260925 04:59:22.135921 23916 heartbeater.cc:464] Registering TS with master...
I20260925 04:59:22.136751 23916 heartbeater.cc:511] Master 127.6.29.190:43841 requested a full tablet report, sending...
I20260925 04:59:22.138417 23459 ts_manager.cc:194] Registered new tserver with Master: f49a44c0fe7b4f168c25cbc628ef4afe (127.6.29.131:36869)
I20260925 04:59:22.139369 23459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.131@KRBTEST.COM'} at 127.6.29.131:36985
I20260925 04:59:22.144686 6262 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:59:22.164963 23918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:22.150017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36378 (local address 127.6.29.190:43841)
0925 04:59:22.150250 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:22.150255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:22.150782 (+ 527us) server_negotiation.cc:407] Connection header received
0925 04:59:22.150979 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:22.150984 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:22.151060 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:22.151178 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:22.152943 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:22.154063 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:22.156150 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:22.156722 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:22.156885 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:22.159436 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:22.159445 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:22.159454 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:59:22.159493 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:22.162073 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:22.162793 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:22.162799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:22.162803 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:22.162960 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:22.163353 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:22.163358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:22.163362 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:22.163714 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:22.164018 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:22.164377 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 04:59:22.164635 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 04:59:22.169296 23459 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36378:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:59:22.176573 23459 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:59:22.200387 23930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:22.189560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44884 (local address 127.6.29.129:36181)
0925 04:59:22.190260 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:22.190267 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:22.190879 (+ 612us) server_negotiation.cc:407] Connection header received
0925 04:59:22.191308 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:22.191314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:22.191497 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:22.191686 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:59:22.192963 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:22.194490 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:22.197208 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:22.198514 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:22.198653 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:22.198909 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:22.199844 (+ 935us) server_negotiation.cc:299] Negotiation successful
0925 04:59:22.200032 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":395,"threads_started":1}
I20260925 04:59:22.201123 23929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:22.189250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:36181 (local address 127.0.0.1:44884)
0925 04:59:22.190735 (+ 1485us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:22.190772 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:22.190957 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:22.191688 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:22.191697 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:22.191728 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:59:22.192716 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:22.192731 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:22.194602 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:22.194608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:22.197055 (+ 2447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:22.197066 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:22.198744 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:22.198755 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:22.198966 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:22.199014 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 04:59:22.200600 (+ 1586us) client_negotiation.cc:240] Negotiation successful
0925 04:59:22.200798 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1241,"thread_start_us":1111,"threads_started":1}
I20260925 04:59:22.201354 23932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:22.191537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56734 (local address 127.6.29.131:36869)
0925 04:59:22.192612 (+ 1075us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:22.192620 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:22.193229 (+ 609us) server_negotiation.cc:407] Connection header received
0925 04:59:22.193332 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:22.193342 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:22.193614 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:22.193814 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:59:22.194600 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:22.195989 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:22.198325 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:22.199745 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:22.199899 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:22.200110 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:22.200839 (+ 729us) server_negotiation.cc:299] Negotiation successful
0925 04:59:22.201023 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":909,"thread_start_us":707,"threads_started":1}
I20260925 04:59:22.202023 23931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:22.191160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.131:36869 (local address 127.0.0.1:56734)
0925 04:59:22.192983 (+ 1823us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:22.193001 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:22.193114 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:22.193911 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:22.193916 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:22.193944 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:59:22.194398 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:22.194409 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:22.196178 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:22.196184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:22.198171 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:22.198183 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:22.200168 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:22.200173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:22.200347 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:22.200377 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 04:59:22.201420 (+ 1043us) client_negotiation.cc:240] Negotiation successful
0925 04:59:22.201571 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1708,"mutex_wait_us":3,"thread_start_us":1545,"threads_started":1}
I20260925 04:59:22.202764 23933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:22.193275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41898 (local address 127.6.29.130:40727)
0925 04:59:22.193784 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:22.193792 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:22.194656 (+ 864us) server_negotiation.cc:407] Connection header received
0925 04:59:22.194906 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:22.194913 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:22.195213 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:22.195410 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:59:22.196054 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:22.197634 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:22.200299 (+ 2665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:22.201746 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:22.201897 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:22.202064 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:22.202411 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 04:59:22.202547 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":245,"threads_started":1}
I20260925 04:59:22.202793 23934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:22.193808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:40727 (local address 127.0.0.1:41898)
0925 04:59:22.194556 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:22.194572 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:22.194711 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:22.195413 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:22.195418 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:22.195444 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:59:22.195842 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:22.195854 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:22.197778 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:22.197783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:22.199577 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:22.199588 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:22.201989 (+ 2401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:22.201998 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:22.202192 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:22.202237 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 04:59:22.202340 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 04:59:22.202468 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":446,"threads_started":1}
I20260925 04:59:22.205334 23583 tablet_service.cc:1467] Processing CreateTablet for tablet 9f436e222fb445a18658c3975d3048be (DEFAULT_TABLE table=test-table [id=5484d6ce14794f7bab86fcdc69695750]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:59:22.206759 23583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f436e222fb445a18658c3975d3048be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:59:22.206666 23851 tablet_service.cc:1467] Processing CreateTablet for tablet 9f436e222fb445a18658c3975d3048be (DEFAULT_TABLE table=test-table [id=5484d6ce14794f7bab86fcdc69695750]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:59:22.206667 23717 tablet_service.cc:1467] Processing CreateTablet for tablet 9f436e222fb445a18658c3975d3048be (DEFAULT_TABLE table=test-table [id=5484d6ce14794f7bab86fcdc69695750]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:59:22.208122 23851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f436e222fb445a18658c3975d3048be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:59:22.208122 23717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f436e222fb445a18658c3975d3048be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:59:22.223487 23935 tablet_bootstrap.cc:492] T 9f436e222fb445a18658c3975d3048be P c6cf5c105516413d8866ea1f6f5b9219: Bootstrap starting.
I20260925 04:59:22.226747 23937 tablet_bootstrap.cc:492] T 9f436e222fb445a18658c3975d3048be P f49a44c0fe7b4f168c25cbc628ef4afe: Bootstrap starting.
I20260925 04:59:22.228232 23935 tablet_bootstrap.cc:654] T 9f436e222fb445a18658c3975d3048be P c6cf5c105516413d8866ea1f6f5b9219: Neither blocks nor log segments found. Creating new log.
I20260925 04:59:22.228348 23936 tablet_bootstrap.cc:492] T 9f436e222fb445a18658c3975d3048be P 279bee6bb6ba4acea53a23125b276eb5: Bootstrap starting.
I20260925 04:59:22.229686 23935 log.cc:824] T 9f436e222fb445a18658c3975d3048be P c6cf5c105516413d8866ea1f6f5b9219: Log is configured to *not* fsync() on all Append() calls
I20260925 04:59:22.230450 23937 tablet_bootstrap.cc:654] T 9f436e222fb445a18658c3975d3048be P f49a44c0fe7b4f168c25cbc628ef4afe: Neither blocks nor log segments found. Creating new log.
I20260925 04:59:22.231832 23937 log.cc:824] T 9f436e222fb445a18658c3975d3048be P f49a44c0fe7b4f168c25cbc628ef4afe: Log is configured to *not* fsync() on all Append() calls
I20260925 04:59:22.231921 23936 tablet_bootstrap.cc:654] T 9f436e222fb445a18658c3975d3048be P 279bee6bb6ba4acea53a23125b276eb5: Neither blocks nor log segments found. Creating new log.
I20260925 04:59:22.232936 23935 tablet_bootstrap.cc:492] T 9f436e222fb445a18658c3975d3048be P c6cf5c105516413d8866ea1f6f5b9219: No bootstrap required, opened a new log
I20260925 04:59:22.233259 23935 ts_tablet_manager.cc:1397] T 9f436e222fb445a18658c3975d3048be P c6cf5c105516413d8866ea1f6f5b9219: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260925 04:59:22.233270 23936 log.cc:824] T 9f436e222fb445a18658c3975d3048be P 279bee6bb6ba4acea53a23125b276eb5: Log is configured to *not* fsync() on all Append() calls
I20260925 04:59:22.235214 23937 tablet_bootstrap.cc:492] T 9f436e222fb445a18658c3975d3048be P f49a44c0fe7b4f168c25cbc628ef4afe: No bootstrap required, opened a new log
I20260925 04:59:22.235539 23937 ts_tablet_manager.cc:1397] T 9f436e222fb445a18658c3975d3048be P f49a44c0fe7b4f168c25cbc628ef4afe: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260925 04:59:22.236325 23936 tablet_bootstrap.cc:492] T 9f436e222fb445a18658c3975d3048be P 279bee6bb6ba4acea53a23125b276eb5: No bootstrap required, opened a new log
I20260925 04:59:22.236580 23936 ts_tablet_manager.cc:1397] T 9f436e222fb445a18658c3975d3048be P 279bee6bb6ba4acea53a23125b276eb5: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260925 04:59:22.241931 23935 raft_consensus.cc:348] T 9f436e222fb445a18658c3975d3048be P c6cf5c105516413d8866ea1f6f5b9219 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "279bee6bb6ba4acea53a23125b276eb5" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 36181 } } peers { permanent_uuid: "f49a44c0fe7b4f168c25cbc628ef4afe" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 36869 } } peers { permanent_uuid: "c6cf5c105516413d8866ea1f6f5b9219" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40727 } }
I20260925 04:59:22.242358 23935 raft_consensus.cc:374] T 9f436e222fb445a18658c3975d3048be P c6cf5c105516413d8866ea1f6f5b9219 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:59:22.242502 23935 raft_consensus.cc:729] T 9f436e222fb445a18658c3975d3048be P c6cf5c105516413d8866ea1f6f5b9219 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6cf5c105516413d8866ea1f6f5b9219, State: Initialized, Role: FOLLOWER
I20260925 04:59:22.243132 23935 consensus_queue.cc:260] T 9f436e222fb445a18658c3975d3048be P c6cf5c105516413d8866ea1f6f5b9219 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "279bee6bb6ba4acea53a23125b276eb5" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 36181 } } peers { permanent_uuid: "f49a44c0fe7b4f168c25cbc628ef4afe" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 36869 } } peers { permanent_uuid: "c6cf5c105516413d8866ea1f6f5b9219" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40727 } }
I20260925 04:59:22.244340 23782 heartbeater.cc:503] Master 127.6.29.190:43841 was elected leader, sending a full tablet report...
I20260925 04:59:22.245082 23935 ts_tablet_manager.cc:1428] T 9f436e222fb445a18658c3975d3048be P c6cf5c105516413d8866ea1f6f5b9219: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260925 04:59:22.246820 23937 raft_consensus.cc:348] T 9f436e222fb445a18658c3975d3048be P f49a44c0fe7b4f168c25cbc628ef4afe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "279bee6bb6ba4acea53a23125b276eb5" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 36181 } } peers { permanent_uuid: "f49a44c0fe7b4f168c25cbc628ef4afe" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 36869 } } peers { permanent_uuid: "c6cf5c105516413d8866ea1f6f5b9219" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40727 } }
I20260925 04:59:22.247390 23937 raft_consensus.cc:374] T 9f436e222fb445a18658c3975d3048be P f49a44c0fe7b4f168c25cbc628ef4afe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:59:22.247548 23937 raft_consensus.cc:729] T 9f436e222fb445a18658c3975d3048be P f49a44c0fe7b4f168c25cbc628ef4afe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f49a44c0fe7b4f168c25cbc628ef4afe, State: Initialized, Role: FOLLOWER
I20260925 04:59:22.247504 23936 raft_consensus.cc:348] T 9f436e222fb445a18658c3975d3048be P 279bee6bb6ba4acea53a23125b276eb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "279bee6bb6ba4acea53a23125b276eb5" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 36181 } } peers { permanent_uuid: "f49a44c0fe7b4f168c25cbc628ef4afe" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 36869 } } peers { permanent_uuid: "c6cf5c105516413d8866ea1f6f5b9219" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40727 } }
I20260925 04:59:22.248018 23936 raft_consensus.cc:374] T 9f436e222fb445a18658c3975d3048be P 279bee6bb6ba4acea53a23125b276eb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:59:22.248157 23936 raft_consensus.cc:729] T 9f436e222fb445a18658c3975d3048be P 279bee6bb6ba4acea53a23125b276eb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 279bee6bb6ba4acea53a23125b276eb5, State: Initialized, Role: FOLLOWER
I20260925 04:59:22.248677 23937 consensus_queue.cc:260] T 9f436e222fb445a18658c3975d3048be P f49a44c0fe7b4f168c25cbc628ef4afe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "279bee6bb6ba4acea53a23125b276eb5" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 36181 } } peers { permanent_uuid: "f49a44c0fe7b4f168c25cbc628ef4afe" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 36869 } } peers { permanent_uuid: "c6cf5c105516413d8866ea1f6f5b9219" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40727 } }
I20260925 04:59:22.248775 23936 consensus_queue.cc:260] T 9f436e222fb445a18658c3975d3048be P 279bee6bb6ba4acea53a23125b276eb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "279bee6bb6ba4acea53a23125b276eb5" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 36181 } } peers { permanent_uuid: "f49a44c0fe7b4f168c25cbc628ef4afe" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 36869 } } peers { permanent_uuid: "c6cf5c105516413d8866ea1f6f5b9219" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40727 } }
I20260925 04:59:22.250792 23937 ts_tablet_manager.cc:1428] T 9f436e222fb445a18658c3975d3048be P f49a44c0fe7b4f168c25cbc628ef4afe: Time spent starting tablet: real 0.015s user 0.005s sys 0.009s
I20260925 04:59:22.251003 23936 ts_tablet_manager.cc:1428] T 9f436e222fb445a18658c3975d3048be P 279bee6bb6ba4acea53a23125b276eb5: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260925 04:59:22.251942 23916 heartbeater.cc:503] Master 127.6.29.190:43841 was elected leader, sending a full tablet report...
W20260925 04:59:22.288221 23649 tablet.cc:2369] T 9f436e222fb445a18658c3975d3048be P 279bee6bb6ba4acea53a23125b276eb5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:59:22.322062 23783 tablet.cc:2369] T 9f436e222fb445a18658c3975d3048be P c6cf5c105516413d8866ea1f6f5b9219: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:59:22.360632 23917 tablet.cc:2369] T 9f436e222fb445a18658c3975d3048be P f49a44c0fe7b4f168c25cbc628ef4afe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:59:22.460083 23941 raft_consensus.cc:482] T 9f436e222fb445a18658c3975d3048be P c6cf5c105516413d8866ea1f6f5b9219 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:59:22.460493 23941 raft_consensus.cc:504] T 9f436e222fb445a18658c3975d3048be P c6cf5c105516413d8866ea1f6f5b9219 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "279bee6bb6ba4acea53a23125b276eb5" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 36181 } } peers { permanent_uuid: "f49a44c0fe7b4f168c25cbc628ef4afe" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 36869 } } peers { permanent_uuid: "c6cf5c105516413d8866ea1f6f5b9219" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40727 } }
I20260925 04:59:22.462188 23941 leader_election.cc:288] T 9f436e222fb445a18658c3975d3048be P c6cf5c105516413d8866ea1f6f5b9219 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 279bee6bb6ba4acea53a23125b276eb5 (127.6.29.129:36181), f49a44c0fe7b4f168c25cbc628ef4afe (127.6.29.131:36869)
I20260925 04:59:22.469156 23930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:22.462736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:35611 (local address 127.6.29.129:36181)
0925 04:59:22.463006 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:22.463015 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:22.463446 (+ 431us) server_negotiation.cc:407] Connection header received
0925 04:59:22.463719 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:22.463724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:22.463835 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:22.463940 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:59:22.464771 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:22.465498 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:22.467294 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:22.468197 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:22.468389 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:22.468468 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:22.468775 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 04:59:22.468902 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 04:59:22.469286 23944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:22.462629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:36181 (local address 127.6.29.130:35611)
0925 04:59:22.463303 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:22.463320 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:22.463526 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:22.464140 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:22.464145 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:22.464170 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:59:22.464630 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:22.464640 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:22.465629 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:22.465637 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:22.467129 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:22.467140 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:22.468439 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:22.468445 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:22.468610 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:22.468634 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 04:59:22.468742 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 04:59:22.468871 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":7,"thread_start_us":384,"threads_started":1}
I20260925 04:59:22.470305 23932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:22.463910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:40805 (local address 127.6.29.131:36869)
0925 04:59:22.464124 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:22.464132 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:22.464397 (+ 265us) server_negotiation.cc:407] Connection header received
0925 04:59:22.464609 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:22.464613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:22.464691 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:22.464828 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:59:22.465383 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:22.466107 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:22.468422 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:22.469192 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:22.469392 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:22.469464 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:22.469918 (+ 454us) server_negotiation.cc:299] Negotiation successful
0925 04:59:22.470049 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 04:59:22.470393 23945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:22.463580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.131:36869 (local address 127.6.29.130:40805)
0925 04:59:22.464288 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:22.464312 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:22.464438 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:22.464850 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:22.464855 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:22.464880 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:59:22.465216 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:22.465224 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:22.466250 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:22.466257 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:22.468261 (+ 2004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:22.468272 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:22.469576 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:22.469585 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:22.469760 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:22.469783 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 04:59:22.469887 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 04:59:22.470005 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":26,"thread_start_us":372,"threads_started":1}
I20260925 04:59:22.470474 23603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f436e222fb445a18658c3975d3048be" candidate_uuid: "c6cf5c105516413d8866ea1f6f5b9219" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "279bee6bb6ba4acea53a23125b276eb5" is_pre_election: true
I20260925 04:59:22.471079 23603 raft_consensus.cc:2454] T 9f436e222fb445a18658c3975d3048be P 279bee6bb6ba4acea53a23125b276eb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6cf5c105516413d8866ea1f6f5b9219 in term 0.
I20260925 04:59:22.471411 23871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f436e222fb445a18658c3975d3048be" candidate_uuid: "c6cf5c105516413d8866ea1f6f5b9219" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f49a44c0fe7b4f168c25cbc628ef4afe" is_pre_election: true
I20260925 04:59:22.471841 23672 leader_election.cc:302] T 9f436e222fb445a18658c3975d3048be P c6cf5c105516413d8866ea1f6f5b9219 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 279bee6bb6ba4acea53a23125b276eb5, c6cf5c105516413d8866ea1f6f5b9219; no voters:
I20260925 04:59:22.471966 23871 raft_consensus.cc:2454] T 9f436e222fb445a18658c3975d3048be P f49a44c0fe7b4f168c25cbc628ef4afe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6cf5c105516413d8866ea1f6f5b9219 in term 0.
I20260925 04:59:22.472278 23941 raft_consensus.cc:2781] T 9f436e222fb445a18658c3975d3048be P c6cf5c105516413d8866ea1f6f5b9219 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:59:22.472438 23941 raft_consensus.cc:482] T 9f436e222fb445a18658c3975d3048be P c6cf5c105516413d8866ea1f6f5b9219 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:59:22.472558 23941 raft_consensus.cc:3037] T 9f436e222fb445a18658c3975d3048be P c6cf5c105516413d8866ea1f6f5b9219 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:59:22.475003 23941 raft_consensus.cc:504] T 9f436e222fb445a18658c3975d3048be P c6cf5c105516413d8866ea1f6f5b9219 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "279bee6bb6ba4acea53a23125b276eb5" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 36181 } } peers { permanent_uuid: "f49a44c0fe7b4f168c25cbc628ef4afe" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 36869 } } peers { permanent_uuid: "c6cf5c105516413d8866ea1f6f5b9219" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40727 } }
I20260925 04:59:22.475600 23941 leader_election.cc:288] T 9f436e222fb445a18658c3975d3048be P c6cf5c105516413d8866ea1f6f5b9219 [CANDIDATE]: Term 1 election: Requested vote from peers 279bee6bb6ba4acea53a23125b276eb5 (127.6.29.129:36181), f49a44c0fe7b4f168c25cbc628ef4afe (127.6.29.131:36869)
I20260925 04:59:22.476135 23603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f436e222fb445a18658c3975d3048be" candidate_uuid: "c6cf5c105516413d8866ea1f6f5b9219" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "279bee6bb6ba4acea53a23125b276eb5"
I20260925 04:59:22.476202 23871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f436e222fb445a18658c3975d3048be" candidate_uuid: "c6cf5c105516413d8866ea1f6f5b9219" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f49a44c0fe7b4f168c25cbc628ef4afe"
I20260925 04:59:22.476364 23603 raft_consensus.cc:3037] T 9f436e222fb445a18658c3975d3048be P 279bee6bb6ba4acea53a23125b276eb5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:59:22.476434 23871 raft_consensus.cc:3037] T 9f436e222fb445a18658c3975d3048be P f49a44c0fe7b4f168c25cbc628ef4afe [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:59:22.478634 23603 raft_consensus.cc:2454] T 9f436e222fb445a18658c3975d3048be P 279bee6bb6ba4acea53a23125b276eb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6cf5c105516413d8866ea1f6f5b9219 in term 1.
I20260925 04:59:22.478709 23871 raft_consensus.cc:2454] T 9f436e222fb445a18658c3975d3048be P f49a44c0fe7b4f168c25cbc628ef4afe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6cf5c105516413d8866ea1f6f5b9219 in term 1.
I20260925 04:59:22.479063 23672 leader_election.cc:302] T 9f436e222fb445a18658c3975d3048be P c6cf5c105516413d8866ea1f6f5b9219 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 279bee6bb6ba4acea53a23125b276eb5, c6cf5c105516413d8866ea1f6f5b9219; no voters:
I20260925 04:59:22.479364 23941 raft_consensus.cc:2781] T 9f436e222fb445a18658c3975d3048be P c6cf5c105516413d8866ea1f6f5b9219 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:59:22.479851 23941 raft_consensus.cc:686] T 9f436e222fb445a18658c3975d3048be P c6cf5c105516413d8866ea1f6f5b9219 [term 1 LEADER]: Becoming Leader. State: Replica: c6cf5c105516413d8866ea1f6f5b9219, State: Running, Role: LEADER
I20260925 04:59:22.480468 23941 consensus_queue.cc:237] T 9f436e222fb445a18658c3975d3048be P c6cf5c105516413d8866ea1f6f5b9219 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "279bee6bb6ba4acea53a23125b276eb5" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 36181 } } peers { permanent_uuid: "f49a44c0fe7b4f168c25cbc628ef4afe" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 36869 } } peers { permanent_uuid: "c6cf5c105516413d8866ea1f6f5b9219" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40727 } }
I20260925 04:59:22.485401 23459 catalog_manager.cc:5495] T 9f436e222fb445a18658c3975d3048be P c6cf5c105516413d8866ea1f6f5b9219 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6cf5c105516413d8866ea1f6f5b9219 (127.6.29.130). New cstate: current_term: 1 leader_uuid: "c6cf5c105516413d8866ea1f6f5b9219" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "279bee6bb6ba4acea53a23125b276eb5" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 36181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f49a44c0fe7b4f168c25cbc628ef4afe" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 36869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6cf5c105516413d8866ea1f6f5b9219" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 40727 } health_report { overall_health: HEALTHY } } }
I20260925 04:59:22.504952 23933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:22.499338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41908 (local address 127.6.29.130:40727)
0925 04:59:22.499550 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:22.499558 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:22.499615 (+ 57us) server_negotiation.cc:407] Connection header received
0925 04:59:22.499908 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:22.499912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:22.500001 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:22.500158 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:59:22.500576 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:22.501295 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:22.502704 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:22.503123 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:22.503258 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:22.503812 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:59:22.504344 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:59:22.504520 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:22.504571 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 04:59:22.504655 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 04:59:22.513382 23871 raft_consensus.cc:1261] T 9f436e222fb445a18658c3975d3048be P f49a44c0fe7b4f168c25cbc628ef4afe [term 1 FOLLOWER]: Refusing update from remote peer c6cf5c105516413d8866ea1f6f5b9219: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:59:22.513382 23603 raft_consensus.cc:1261] T 9f436e222fb445a18658c3975d3048be P 279bee6bb6ba4acea53a23125b276eb5 [term 1 FOLLOWER]: Refusing update from remote peer c6cf5c105516413d8866ea1f6f5b9219: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:59:22.514362 23946 consensus_queue.cc:1035] T 9f436e222fb445a18658c3975d3048be P c6cf5c105516413d8866ea1f6f5b9219 [LEADER]: Connected to new peer: Peer: permanent_uuid: "279bee6bb6ba4acea53a23125b276eb5" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 36181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:59:22.514752 23941 consensus_queue.cc:1035] T 9f436e222fb445a18658c3975d3048be P c6cf5c105516413d8866ea1f6f5b9219 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f49a44c0fe7b4f168c25cbc628ef4afe" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 36869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:59:22.526489 23949 mvcc.cc:205] Tried to move back new op lower bound from 7333119436844453888 to 7333119436724199424. Current Snapshot: MvccSnapshot[applied={T|T < 7333119436844453888}]
I20260925 04:59:22.527755 23950 mvcc.cc:205] Tried to move back new op lower bound from 7333119436844453888 to 7333119436724199424. Current Snapshot: MvccSnapshot[applied={T|T < 7333119436844453888}]
I20260925 04:59:22.534089 23951 mvcc.cc:205] Tried to move back new op lower bound from 7333119436844453888 to 7333119436724199424. Current Snapshot: MvccSnapshot[applied={T|T < 7333119436844453888}]
I20260925 04:59:22.543365 23459 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:59:22.543670 23459 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:59:22.547861 23459 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0553d55d329d47d38aa10e9dbe3b07a0: Sending DeleteTablet for 3 replicas of tablet 9f436e222fb445a18658c3975d3048be
I20260925 04:59:22.549103 23583 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f436e222fb445a18658c3975d3048be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:59:22 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:44884
I20260925 04:59:22.549242 23851 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f436e222fb445a18658c3975d3048be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:59:22 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:56734
I20260925 04:59:22.550097 23959 tablet_replica.cc:330] T 9f436e222fb445a18658c3975d3048be P 279bee6bb6ba4acea53a23125b276eb5: stopping tablet replica
I20260925 04:59:22.550290 23717 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f436e222fb445a18658c3975d3048be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:59:22 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:41898
I20260925 04:59:22.550596 23959 raft_consensus.cc:2229] T 9f436e222fb445a18658c3975d3048be P 279bee6bb6ba4acea53a23125b276eb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:59:22.550793 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 23520
I20260925 04:59:22.551198 23959 raft_consensus.cc:2258] T 9f436e222fb445a18658c3975d3048be P 279bee6bb6ba4acea53a23125b276eb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:59:22.553644 23959 ts_tablet_manager.cc:1905] T 9f436e222fb445a18658c3975d3048be P 279bee6bb6ba4acea53a23125b276eb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:59:22.563299 23959 ts_tablet_manager.cc:1918] T 9f436e222fb445a18658c3975d3048be P 279bee6bb6ba4acea53a23125b276eb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:59:22.563596 23959 log.cc:1196] T 9f436e222fb445a18658c3975d3048be P 279bee6bb6ba4acea53a23125b276eb5: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-0/wal/wals/9f436e222fb445a18658c3975d3048be
I20260925 04:59:22.564340 23959 ts_tablet_manager.cc:1939] T 9f436e222fb445a18658c3975d3048be P 279bee6bb6ba4acea53a23125b276eb5: Deleting consensus metadata
I20260925 04:59:22.566856 23445 catalog_manager.cc:4841] TS 279bee6bb6ba4acea53a23125b276eb5 (127.6.29.129:36181): tablet 9f436e222fb445a18658c3975d3048be (table test-table [id=5484d6ce14794f7bab86fcdc69695750]) successfully deleted
I20260925 04:59:22.573434 23961 tablet_replica.cc:330] T 9f436e222fb445a18658c3975d3048be P c6cf5c105516413d8866ea1f6f5b9219: stopping tablet replica
I20260925 04:59:22.576735 23960 tablet_replica.cc:330] T 9f436e222fb445a18658c3975d3048be P f49a44c0fe7b4f168c25cbc628ef4afe: stopping tablet replica
I20260925 04:59:22.577270 23960 raft_consensus.cc:2229] T 9f436e222fb445a18658c3975d3048be P f49a44c0fe7b4f168c25cbc628ef4afe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:59:22.577860 23960 raft_consensus.cc:2258] T 9f436e222fb445a18658c3975d3048be P f49a44c0fe7b4f168c25cbc628ef4afe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:59:22.578836 23961 raft_consensus.cc:2229] T 9f436e222fb445a18658c3975d3048be P c6cf5c105516413d8866ea1f6f5b9219 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:59:22.579591 23961 raft_consensus.cc:2258] T 9f436e222fb445a18658c3975d3048be P c6cf5c105516413d8866ea1f6f5b9219 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:59:22.582198 23961 ts_tablet_manager.cc:1905] T 9f436e222fb445a18658c3975d3048be P c6cf5c105516413d8866ea1f6f5b9219: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:59:22.583094 23960 ts_tablet_manager.cc:1905] T 9f436e222fb445a18658c3975d3048be P f49a44c0fe7b4f168c25cbc628ef4afe: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 04:59:22 dist-test-slave-xzjr krb5kdc[23414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.129@KRBTEST.COM
I20260925 04:59:22.590018 23961 ts_tablet_manager.cc:1918] T 9f436e222fb445a18658c3975d3048be P c6cf5c105516413d8866ea1f6f5b9219: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:59:22.590278 23961 log.cc:1196] T 9f436e222fb445a18658c3975d3048be P c6cf5c105516413d8866ea1f6f5b9219: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-1/wal/wals/9f436e222fb445a18658c3975d3048be
I20260925 04:59:22.590922 23961 ts_tablet_manager.cc:1939] T 9f436e222fb445a18658c3975d3048be P c6cf5c105516413d8866ea1f6f5b9219: Deleting consensus metadata
I20260925 04:59:22.592756 23960 ts_tablet_manager.cc:1918] T 9f436e222fb445a18658c3975d3048be P f49a44c0fe7b4f168c25cbc628ef4afe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:59:22.592868 23445 catalog_manager.cc:4841] TS c6cf5c105516413d8866ea1f6f5b9219 (127.6.29.130:40727): tablet 9f436e222fb445a18658c3975d3048be (table test-table [id=5484d6ce14794f7bab86fcdc69695750]) successfully deleted
I20260925 04:59:22.593007 23960 log.cc:1196] T 9f436e222fb445a18658c3975d3048be P f49a44c0fe7b4f168c25cbc628ef4afe: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790312053666503-6262-0/minicluster-data/ts-2/wal/wals/9f436e222fb445a18658c3975d3048be
I20260925 04:59:22.593490 23960 ts_tablet_manager.cc:1939] T 9f436e222fb445a18658c3975d3048be P f49a44c0fe7b4f168c25cbc628ef4afe: Deleting consensus metadata
I20260925 04:59:22.594144 23930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:22.552406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44886 (local address 127.6.29.129:36181)
0925 04:59:22.552568 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:22.552574 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:22.573049 (+ 20475us) server_negotiation.cc:407] Connection header received
0925 04:59:22.573968 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:22.573975 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:22.574089 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:22.574225 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:22.576638 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:22.577785 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:22.579929 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:22.580585 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:22.580711 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:22.586413 (+ 5702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:22.586434 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:22.586451 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:59:22.586504 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:22.590464 (+ 3960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:22.591394 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:22.591403 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:22.591406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:22.591565 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:22.592089 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:22.592093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:22.592096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:22.592549 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:22.592845 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:22.593502 (+ 657us) server_negotiation.cc:299] Negotiation successful
0925 04:59:22.593742 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 04:59:22.595468 23445 catalog_manager.cc:4841] TS f49a44c0fe7b4f168c25cbc628ef4afe (127.6.29.131:36869): tablet 9f436e222fb445a18658c3975d3048be (table test-table [id=5484d6ce14794f7bab86fcdc69695750]) successfully deleted
I20260925 04:59:22.595559 23643 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:59:22.688993 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 23520
I20260925 04:59:22.707647 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 23654
Sep 25 04:59:22 dist-test-slave-xzjr krb5kdc[23414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.130@KRBTEST.COM
I20260925 04:59:22.726899 23933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:22.708832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41924 (local address 127.6.29.130:40727)
0925 04:59:22.708988 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:22.708993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:22.709007 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:59:22.709059 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:22.709063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:22.709182 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:22.709260 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:22.710939 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:22.711887 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:22.713797 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:22.714324 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:22.714431 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:22.720142 (+ 5711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:22.720166 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:22.720185 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:59:22.720243 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:22.723676 (+ 3433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:22.724532 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:22.724541 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:22.724543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:22.724714 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:22.725145 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:22.725154 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:22.725158 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:22.725567 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:22.726012 (+ 445us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:22.726235 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 04:59:22.726481 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 04:59:22.727859 23777 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:59:22.843292 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 23654
I20260925 04:59:22.862931 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 23788
Sep 25 04:59:22 dist-test-slave-xzjr krb5kdc[23414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.131@KRBTEST.COM
I20260925 04:59:22.878021 23932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:22.863681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56738 (local address 127.6.29.131:36869)
0925 04:59:22.863902 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:22.863910 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:22.863966 (+ 56us) server_negotiation.cc:407] Connection header received
0925 04:59:22.864194 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:22.864199 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:22.864282 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:22.864409 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:22.865715 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:22.866432 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:22.867656 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:22.868149 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:22.868316 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:22.872640 (+ 4324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:22.872654 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:22.872687 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:59:22.872722 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:22.875199 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:22.875915 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:22.875926 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:22.875932 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:22.876113 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:22.876493 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:22.876496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:22.876498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:22.876855 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:22.877156 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:22.877463 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 04:59:22.877682 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260925 04:59:22.878660 23911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:59:22.969345 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 23788
I20260925 04:59:22.986284 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 23427
I20260925 04:59:22.987402 23489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:59:23.099927 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 23427
2026-09-25T04:59:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4595 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T04:59:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:59:23Z Disabled control of system clock
I20260925 04:59:23.182528 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:35643
--webserver_interface=127.6.29.190
--webserver_port=0
--builtin_ntp_servers=127.6.29.148:40879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:35643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 04:59:23.515791 23970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:59:23.516104 23970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:59:23.516196 23970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:59:23.523619 23970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:59:23.523731 23970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:59:23.523789 23970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:59:23.523844 23970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:59:23.523888 23970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:59:23.533166 23970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:40879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:35643
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:35643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:59:23.534441 23970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:59:23.536273 23970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:59:23.548069 23975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:23.548127 23978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:23.548094 23976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:59:23.549096 23970 server_base.cc:1034] running on GCE node
I20260925 04:59:23.549851 23970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:59:23.551196 23970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:59:23.552438 23970 hybrid_clock.cc:648] HybridClock initialized: now 1790312363552367 us; error 55 us; skew 500 ppm
I20260925 04:59:23.552846 23970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 04:59:23.560698 23970 webserver.cc:460] Webserver started at http://127.6.29.190:40905/ using document root <none> and password file <none>
I20260925 04:59:23.561421 23970 fs_manager.cc:359] Metadata directory not provided
I20260925 04:59:23.561538 23970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:59:23.561889 23970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:59:23.564997 23970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "18948d6de4014a7a8eec5ff6122369ec"
format_stamp: "Formatted at 2026-09-25 04:59:23 on dist-test-slave-xzjr"
server_key: "b2ee33524b9c095e2df65aebe350aeb7"
server_key_iv: "72c0e80e4b21acdf66292772c69153db"
server_key_version: "encryptionkey@0"
I20260925 04:59:23.565848 23970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "18948d6de4014a7a8eec5ff6122369ec"
format_stamp: "Formatted at 2026-09-25 04:59:23 on dist-test-slave-xzjr"
server_key: "b2ee33524b9c095e2df65aebe350aeb7"
server_key_iv: "72c0e80e4b21acdf66292772c69153db"
server_key_version: "encryptionkey@0"
I20260925 04:59:23.571094 23970 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 04:59:23.574596 23984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:23.576047 23970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 04:59:23.576238 23970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "18948d6de4014a7a8eec5ff6122369ec"
format_stamp: "Formatted at 2026-09-25 04:59:23 on dist-test-slave-xzjr"
server_key: "b2ee33524b9c095e2df65aebe350aeb7"
server_key_iv: "72c0e80e4b21acdf66292772c69153db"
server_key_version: "encryptionkey@0"
I20260925 04:59:23.576467 23970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:59:23.597450 23970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:59:23.598412 23970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:59:23.598733 23970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:59:23.617404 23970 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:35643
I20260925 04:59:23.617424 24035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:35643 every 8 connection(s)
I20260925 04:59:23.618808 23970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:59:23.622659 24036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:59:23.626974 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 23970
I20260925 04:59:23.627179 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 04:59:23.627707 6262 external_mini_cluster.cc:1428] Setting key 98c4197861b6237407dc70c1c97a849d
I20260925 04:59:23.634356 24036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18948d6de4014a7a8eec5ff6122369ec: Bootstrap starting.
I20260925 04:59:23.637938 24036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18948d6de4014a7a8eec5ff6122369ec: Neither blocks nor log segments found. Creating new log.
I20260925 04:59:23.639189 24036 log.cc:824] T 00000000000000000000000000000000 P 18948d6de4014a7a8eec5ff6122369ec: Log is configured to *not* fsync() on all Append() calls
I20260925 04:59:23.642138 24036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18948d6de4014a7a8eec5ff6122369ec: No bootstrap required, opened a new log
I20260925 04:59:23.642390 24039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:23.629730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50654 (local address 127.6.29.190:35643)
0925 04:59:23.630778 (+ 1048us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:23.630789 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:23.630830 (+ 41us) server_negotiation.cc:407] Connection header received
0925 04:59:23.632870 (+ 2040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:23.632894 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:23.633365 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:23.633685 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:23.634335 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:23.635649 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:23.637649 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:23.638440 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:23.638563 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:23.639926 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:23.639935 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:23.639953 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:59:23.640574 (+ 621us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:23.640668 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:59:23.640693 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:23.640805 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:23.641136 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 04:59:23.641413 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":425,"threads_started":1}
I20260925 04:59:23.649887 24036 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18948d6de4014a7a8eec5ff6122369ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18948d6de4014a7a8eec5ff6122369ec" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 35643 } }
I20260925 04:59:23.650324 24036 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18948d6de4014a7a8eec5ff6122369ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:59:23.650451 24036 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18948d6de4014a7a8eec5ff6122369ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18948d6de4014a7a8eec5ff6122369ec, State: Initialized, Role: FOLLOWER
I20260925 04:59:23.651093 24036 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18948d6de4014a7a8eec5ff6122369ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18948d6de4014a7a8eec5ff6122369ec" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 35643 } }
I20260925 04:59:23.651347 24036 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18948d6de4014a7a8eec5ff6122369ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:59:23.651479 24036 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18948d6de4014a7a8eec5ff6122369ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:59:23.651657 24036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18948d6de4014a7a8eec5ff6122369ec [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:59:23.653858 24036 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18948d6de4014a7a8eec5ff6122369ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18948d6de4014a7a8eec5ff6122369ec" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 35643 } }
I20260925 04:59:23.654409 24036 leader_election.cc:302] T 00000000000000000000000000000000 P 18948d6de4014a7a8eec5ff6122369ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18948d6de4014a7a8eec5ff6122369ec; no voters:
I20260925 04:59:23.655112 24036 leader_election.cc:288] T 00000000000000000000000000000000 P 18948d6de4014a7a8eec5ff6122369ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:59:23.655273 24041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18948d6de4014a7a8eec5ff6122369ec [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:59:23.656030 24041 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18948d6de4014a7a8eec5ff6122369ec [term 1 LEADER]: Becoming Leader. State: Replica: 18948d6de4014a7a8eec5ff6122369ec, State: Running, Role: LEADER
I20260925 04:59:23.656577 24041 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18948d6de4014a7a8eec5ff6122369ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18948d6de4014a7a8eec5ff6122369ec" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 35643 } }
I20260925 04:59:23.657218 24036 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18948d6de4014a7a8eec5ff6122369ec [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:59:23.659267 24042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18948d6de4014a7a8eec5ff6122369ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18948d6de4014a7a8eec5ff6122369ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18948d6de4014a7a8eec5ff6122369ec" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 35643 } } }
I20260925 04:59:23.659833 24042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18948d6de4014a7a8eec5ff6122369ec [sys.catalog]: This master's current role is: LEADER
I20260925 04:59:23.660035 24043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18948d6de4014a7a8eec5ff6122369ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18948d6de4014a7a8eec5ff6122369ec. Latest consensus state: current_term: 1 leader_uuid: "18948d6de4014a7a8eec5ff6122369ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18948d6de4014a7a8eec5ff6122369ec" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 35643 } } }
I20260925 04:59:23.660403 24043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18948d6de4014a7a8eec5ff6122369ec [sys.catalog]: This master's current role is: LEADER
I20260925 04:59:23.661540 24046 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:59:23.668201 24046 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:59:23.677501 24046 catalog_manager.cc:1347] Generated new cluster ID: 4e124e84d6c640f391327b07743028bb
I20260925 04:59:23.677637 24046 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:59:23.685690 24046 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:59:23.686928 24046 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:59:23.692975 24046 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18948d6de4014a7a8eec5ff6122369ec: Generated new TSK 0
I20260925 04:59:23.693921 24046 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:59:23.709374 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.129:0
--local_ip_for_outbound_sockets=127.6.29.129
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:35643
--builtin_ntp_servers=127.6.29.148:40879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 04:59:24.046947 24060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:59:24.047251 24060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:59:24.047401 24060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:59:24.054728 24060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:59:24.054821 24060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:59:24.054989 24060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.129
I20260925 04:59:24.064504 24060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:40879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:35643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.129
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:59:24.065811 24060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:59:24.067569 24060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:59:24.080753 24066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:24.080907 24068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:24.080940 24065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:59:24.082285 24060 server_base.cc:1034] running on GCE node
I20260925 04:59:24.082942 24060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:59:24.084026 24060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:59:24.085310 24060 hybrid_clock.cc:648] HybridClock initialized: now 1790312364085218 us; error 81 us; skew 500 ppm
I20260925 04:59:24.085608 24060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 04:59:24.088197 24060 webserver.cc:460] Webserver started at http://127.6.29.129:42151/ using document root <none> and password file <none>
I20260925 04:59:24.088891 24060 fs_manager.cc:359] Metadata directory not provided
I20260925 04:59:24.088984 24060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:59:24.089270 24060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:59:24.092231 24060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-0/data/instance:
uuid: "7da9920def4049d0a3b22aa9c572fae3"
format_stamp: "Formatted at 2026-09-25 04:59:24 on dist-test-slave-xzjr"
server_key: "15332221adda6a0a5dc278d1e49e21dc"
server_key_iv: "2258f884d63a4ad8d22dbc2b2f79db66"
server_key_version: "encryptionkey@0"
I20260925 04:59:24.092958 24060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance:
uuid: "7da9920def4049d0a3b22aa9c572fae3"
format_stamp: "Formatted at 2026-09-25 04:59:24 on dist-test-slave-xzjr"
server_key: "15332221adda6a0a5dc278d1e49e21dc"
server_key_iv: "2258f884d63a4ad8d22dbc2b2f79db66"
server_key_version: "encryptionkey@0"
I20260925 04:59:24.098089 24060 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 04:59:24.101809 24074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:24.103434 24060 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260925 04:59:24.103585 24060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-0/wal
uuid: "7da9920def4049d0a3b22aa9c572fae3"
format_stamp: "Formatted at 2026-09-25 04:59:24 on dist-test-slave-xzjr"
server_key: "15332221adda6a0a5dc278d1e49e21dc"
server_key_iv: "2258f884d63a4ad8d22dbc2b2f79db66"
server_key_version: "encryptionkey@0"
I20260925 04:59:24.103843 24060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:59:24.138720 24060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:59:24.139547 24060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:59:24.139839 24060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:59:24.140838 24060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:59:24.142400 24060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:59:24.142486 24060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:24.142560 24060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:59:24.142622 24060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:24.178627 24060 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.129:38391
I20260925 04:59:24.178704 24186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.129:38391 every 8 connection(s)
I20260925 04:59:24.180012 24060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
I20260925 04:59:24.184161 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 24060
I20260925 04:59:24.184340 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance
I20260925 04:59:24.184988 6262 external_mini_cluster.cc:1428] Setting key 3f19080b87f0402077e852fbceb40bf6
I20260925 04:59:24.188098 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.130:0
--local_ip_for_outbound_sockets=127.6.29.130
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:35643
--builtin_ntp_servers=127.6.29.148:40879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 04:59:24.194547 24189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:24.182526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:56043 (local address 127.6.29.190:35643)
0925 04:59:24.182978 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:24.182983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:24.184180 (+ 1197us) server_negotiation.cc:407] Connection header received
0925 04:59:24.186446 (+ 2266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:24.186451 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:24.186548 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:24.186663 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:24.188111 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:24.188941 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:24.191437 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:24.191920 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:24.192074 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:24.193803 (+ 1729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:24.193807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:24.193811 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:59:24.193830 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:24.193867 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:59:24.193878 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:24.193950 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:24.194215 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 04:59:24.194292 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":253,"threads_started":1}
I20260925 04:59:24.196117 24190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:24.183017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:35643 (local address 127.6.29.129:56043)
0925 04:59:24.184001 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:24.184048 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:24.186169 (+ 2121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:24.186908 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:24.186918 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:24.187055 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:59:24.187931 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:24.187961 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:24.189106 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:24.189114 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:24.191289 (+ 2175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:24.191307 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:24.192152 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:24.192162 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:24.192527 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:24.193438 (+ 911us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:59:24.193444 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:59:24.193503 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:59:24.193616 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:59:24.193977 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:59:24.193990 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:59:24.194002 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 04:59:24.194184 (+ 182us) client_negotiation.cc:240] Negotiation successful
0925 04:59:24.194633 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":263,"threads_started":1}
I20260925 04:59:24.197994 24187 heartbeater.cc:347] Connected to a master server at 127.6.29.190:35643
I20260925 04:59:24.198328 24187 heartbeater.cc:464] Registering TS with master...
I20260925 04:59:24.199085 24187 heartbeater.cc:511] Master 127.6.29.190:35643 requested a full tablet report, sending...
I20260925 04:59:24.201375 24001 ts_manager.cc:194] Registered new tserver with Master: 7da9920def4049d0a3b22aa9c572fae3 (127.6.29.129:38391)
I20260925 04:59:24.203531 24001 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.29.129:56043
W20260925 04:59:24.527148 24191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:59:24.527453 24191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:59:24.527540 24191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:59:24.534958 24191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:59:24.535076 24191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:59:24.535184 24191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.130
I20260925 04:59:24.544751 24191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:40879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:35643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.130
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:59:24.546094 24191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:59:24.547987 24191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:59:24.561173 24196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:24.561228 24197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:24.561357 24199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:59:24.562026 24191 server_base.cc:1034] running on GCE node
I20260925 04:59:24.562681 24191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:59:24.563669 24191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:59:24.564934 24191 hybrid_clock.cc:648] HybridClock initialized: now 1790312364564873 us; error 49 us; skew 500 ppm
I20260925 04:59:24.565322 24191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 04:59:24.567664 24191 webserver.cc:460] Webserver started at http://127.6.29.130:43613/ using document root <none> and password file <none>
I20260925 04:59:24.568408 24191 fs_manager.cc:359] Metadata directory not provided
I20260925 04:59:24.568532 24191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:59:24.568851 24191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:59:24.571822 24191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-1/data/instance:
uuid: "671dcaacd498452a8dcfe825783fdbe4"
format_stamp: "Formatted at 2026-09-25 04:59:24 on dist-test-slave-xzjr"
server_key: "e0f38ee7f86cf5099721db1512c13bf8"
server_key_iv: "6bbf09836223c66ba6cbdf6f747bc50a"
server_key_version: "encryptionkey@0"
I20260925 04:59:24.572677 24191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance:
uuid: "671dcaacd498452a8dcfe825783fdbe4"
format_stamp: "Formatted at 2026-09-25 04:59:24 on dist-test-slave-xzjr"
server_key: "e0f38ee7f86cf5099721db1512c13bf8"
server_key_iv: "6bbf09836223c66ba6cbdf6f747bc50a"
server_key_version: "encryptionkey@0"
I20260925 04:59:24.577968 24191 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260925 04:59:24.581791 24205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:24.583230 24191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:59:24.583422 24191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-1/wal
uuid: "671dcaacd498452a8dcfe825783fdbe4"
format_stamp: "Formatted at 2026-09-25 04:59:24 on dist-test-slave-xzjr"
server_key: "e0f38ee7f86cf5099721db1512c13bf8"
server_key_iv: "6bbf09836223c66ba6cbdf6f747bc50a"
server_key_version: "encryptionkey@0"
I20260925 04:59:24.583657 24191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:59:24.606859 24191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:59:24.607744 24191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:59:24.608134 24191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:59:24.609179 24191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:59:24.610662 24191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:59:24.610778 24191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:24.610908 24191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:59:24.610999 24191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:24.644037 24191 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.130:39399
I20260925 04:59:24.644094 24317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.130:39399 every 8 connection(s)
I20260925 04:59:24.645614 24191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
I20260925 04:59:24.646857 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 24191
I20260925 04:59:24.647055 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance
I20260925 04:59:24.647665 6262 external_mini_cluster.cc:1428] Setting key cad9a4cdd246df23bd0bf13f38eb11d2
I20260925 04:59:24.650943 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.131:0
--local_ip_for_outbound_sockets=127.6.29.131
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:35643
--builtin_ntp_servers=127.6.29.148:40879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 04:59:24.661932 24189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:24.648722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:38595 (local address 127.6.29.190:35643)
0925 04:59:24.649658 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:24.649667 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:24.650478 (+ 811us) server_negotiation.cc:407] Connection header received
0925 04:59:24.653486 (+ 3008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:24.653491 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:24.653561 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:24.653657 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:24.655159 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:24.655989 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:24.658662 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:24.659098 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:24.659233 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:24.660961 (+ 1728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:24.660969 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:24.660975 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:59:24.661005 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:24.661050 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:59:24.661071 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:24.661188 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:24.661388 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 04:59:24.661509 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":803}
I20260925 04:59:24.663005 24320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:24.649218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:35643 (local address 127.6.29.130:38595)
0925 04:59:24.650323 (+ 1105us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:24.650377 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:24.653148 (+ 2771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:24.653948 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:24.653962 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:24.654094 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:59:24.654929 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:24.654955 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:24.656160 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:24.656168 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:24.658487 (+ 2319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:24.658508 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:24.659361 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:24.659371 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:24.659732 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:24.660587 (+ 855us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:59:24.660590 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:59:24.660635 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:59:24.660734 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:59:24.661180 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:59:24.661187 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:59:24.661194 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 04:59:24.661348 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 04:59:24.661670 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":383,"threads_started":1}
I20260925 04:59:24.664584 24318 heartbeater.cc:347] Connected to a master server at 127.6.29.190:35643
I20260925 04:59:24.664916 24318 heartbeater.cc:464] Registering TS with master...
I20260925 04:59:24.665674 24318 heartbeater.cc:511] Master 127.6.29.190:35643 requested a full tablet report, sending...
I20260925 04:59:24.667150 24001 ts_manager.cc:194] Registered new tserver with Master: 671dcaacd498452a8dcfe825783fdbe4 (127.6.29.130:39399)
I20260925 04:59:24.668099 24001 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.29.130:38595
W20260925 04:59:24.992458 24321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:59:24.992749 24321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:59:24.992887 24321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:59:25.000293 24321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:59:25.000401 24321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:59:25.000559 24321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.131
I20260925 04:59:25.010028 24321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:40879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:35643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.131
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:59:25.011413 24321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:59:25.013273 24321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:59:25.027297 24327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:25.027434 24329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:25.027302 24326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:59:25.028077 24321 server_base.cc:1034] running on GCE node
I20260925 04:59:25.028803 24321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:59:25.029819 24321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:59:25.031072 24321 hybrid_clock.cc:648] HybridClock initialized: now 1790312365031011 us; error 58 us; skew 500 ppm
I20260925 04:59:25.031414 24321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 04:59:25.033814 24321 webserver.cc:460] Webserver started at http://127.6.29.131:35063/ using document root <none> and password file <none>
I20260925 04:59:25.034524 24321 fs_manager.cc:359] Metadata directory not provided
I20260925 04:59:25.034652 24321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:59:25.034979 24321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:59:25.037941 24321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-2/data/instance:
uuid: "c6b50966ef604bd38cfe92d83979564e"
format_stamp: "Formatted at 2026-09-25 04:59:25 on dist-test-slave-xzjr"
server_key: "ea0c605f3407e251b38a10491ad6dabd"
server_key_iv: "c01868c0648a908ee7718ad1f2ff6bc7"
server_key_version: "encryptionkey@0"
I20260925 04:59:25.038774 24321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance:
uuid: "c6b50966ef604bd38cfe92d83979564e"
format_stamp: "Formatted at 2026-09-25 04:59:25 on dist-test-slave-xzjr"
server_key: "ea0c605f3407e251b38a10491ad6dabd"
server_key_iv: "c01868c0648a908ee7718ad1f2ff6bc7"
server_key_version: "encryptionkey@0"
I20260925 04:59:25.044010 24321 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 04:59:25.047535 24335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:25.048993 24321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:59:25.049171 24321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-2/wal
uuid: "c6b50966ef604bd38cfe92d83979564e"
format_stamp: "Formatted at 2026-09-25 04:59:25 on dist-test-slave-xzjr"
server_key: "ea0c605f3407e251b38a10491ad6dabd"
server_key_iv: "c01868c0648a908ee7718ad1f2ff6bc7"
server_key_version: "encryptionkey@0"
I20260925 04:59:25.049394 24321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:59:25.081854 24321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:59:25.082757 24321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:59:25.083057 24321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:59:25.084125 24321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:59:25.085644 24321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:59:25.085737 24321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:25.085887 24321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:59:25.085978 24321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:25.120018 24321 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.131:43619
I20260925 04:59:25.120110 24447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.131:43619 every 8 connection(s)
I20260925 04:59:25.121364 24321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
I20260925 04:59:25.129962 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 24321
I20260925 04:59:25.130167 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance
I20260925 04:59:25.130650 6262 external_mini_cluster.cc:1428] Setting key c0264a751e2dc87b99a03a6330fcf097
I20260925 04:59:25.134688 24189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:25.124039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:42179 (local address 127.6.29.190:35643)
0925 04:59:25.124260 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:25.124267 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:25.125692 (+ 1425us) server_negotiation.cc:407] Connection header received
0925 04:59:25.127504 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:25.127509 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:25.127595 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:25.127695 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:25.128854 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:25.129719 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:25.131563 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:25.132280 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:25.132428 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:25.133970 (+ 1542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:25.133975 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:25.133978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:59:25.133996 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:25.134030 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:59:25.134042 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:25.134116 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:25.134354 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 04:59:25.134434 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 04:59:25.135692 24450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:25.124609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:35643 (local address 127.6.29.131:42179)
0925 04:59:25.125524 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:25.125567 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:25.127219 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:25.127954 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:25.127962 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:25.128054 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:59:25.128685 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:25.128702 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:25.129857 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:25.129860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:25.131425 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:25.131437 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:25.132505 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:25.132511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:25.132808 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:25.133668 (+ 860us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:59:25.133670 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:59:25.133713 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:59:25.133799 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:59:25.134131 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:59:25.134139 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:59:25.134145 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 04:59:25.134298 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 04:59:25.134596 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":284,"threads_started":1}
I20260925 04:59:25.137166 24448 heartbeater.cc:347] Connected to a master server at 127.6.29.190:35643
I20260925 04:59:25.137441 24448 heartbeater.cc:464] Registering TS with master...
I20260925 04:59:25.138164 24448 heartbeater.cc:511] Master 127.6.29.190:35643 requested a full tablet report, sending...
I20260925 04:59:25.139498 24001 ts_manager.cc:194] Registered new tserver with Master: c6b50966ef604bd38cfe92d83979564e (127.6.29.131:43619)
I20260925 04:59:25.140321 24001 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.29.131:42179
I20260925 04:59:25.145998 6262 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:59:25.156718 24189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:25.150352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50660 (local address 127.6.29.190:35643)
0925 04:59:25.150622 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:25.150628 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:25.150955 (+ 327us) server_negotiation.cc:407] Connection header received
0925 04:59:25.151093 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:25.151096 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:25.151210 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:25.151295 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:25.151891 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:25.152635 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:25.154227 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:25.154726 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:25.154826 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:25.155932 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:25.155940 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:25.155948 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:59:25.155975 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:25.156018 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:59:25.156034 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:25.156133 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:25.156312 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 04:59:25.156391 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 04:59:25.160848 24001 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50660:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:59:25.168874 24001 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:59:25.206578 24462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:25.181940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58604 (local address 127.6.29.130:39399)
0925 04:59:25.197866 (+ 15926us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:25.197888 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:25.197915 (+ 27us) server_negotiation.cc:407] Connection header received
0925 04:59:25.198004 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:25.198009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:25.198247 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:25.198480 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:59:25.199153 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:25.200471 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:25.203505 (+ 3034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:25.205017 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:25.205141 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:25.205398 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:25.206031 (+ 633us) server_negotiation.cc:299] Negotiation successful
0925 04:59:25.206218 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15778,"thread_start_us":15685,"threads_started":1}
I20260925 04:59:25.206588 24460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:25.181403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:39399 (local address 127.0.0.1:58604)
0925 04:59:25.196039 (+ 14636us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:25.196104 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:25.196400 (+ 296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:25.198500 (+ 2100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:25.198508 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:25.198543 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:59:25.198952 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:25.198967 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:25.200661 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:25.200668 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:25.203336 (+ 2668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:25.203347 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:25.205537 (+ 2190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:25.205543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:25.205761 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:25.205811 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 04:59:25.205962 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 04:59:25.206174 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14238,"thread_start_us":14070,"threads_started":1}
I20260925 04:59:25.206797 24187 heartbeater.cc:503] Master 127.6.29.190:35643 was elected leader, sending a full tablet report...
I20260925 04:59:25.209592 24464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:25.181752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.131:43619 (local address 127.0.0.1:50752)
0925 04:59:25.200267 (+ 18515us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:25.200288 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:25.200403 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:25.201981 (+ 1578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:25.201986 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:25.202013 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:59:25.202443 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:25.202454 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:25.204737 (+ 2283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:25.204741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:25.206494 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:25.206515 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:25.208849 (+ 2334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:25.208855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:25.209024 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:25.209055 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 04:59:25.209147 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 04:59:25.209283 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18370,"thread_start_us":20971,"threads_started":1}
I20260925 04:59:25.209703 24463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:25.182016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50752 (local address 127.6.29.131:43619)
0925 04:59:25.201024 (+ 19008us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:25.201034 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:25.201070 (+ 36us) server_negotiation.cc:407] Connection header received
0925 04:59:25.201167 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:25.201175 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:25.201465 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:25.201678 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:59:25.202666 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:25.204561 (+ 1895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:25.206765 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:25.208315 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:25.208439 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:25.208725 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:25.209233 (+ 508us) server_negotiation.cc:299] Negotiation successful
0925 04:59:25.209396 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18801,"thread_start_us":19471,"threads_started":1}
I20260925 04:59:25.211067 24253 tablet_service.cc:1467] Processing CreateTablet for tablet 207eb00544e74e1caa151902b5d30381 (DEFAULT_TABLE table=test-table [id=047be7fdd2614d1399c3f717584e17d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:59:25.212366 24253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 207eb00544e74e1caa151902b5d30381. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:59:25.213158 24383 tablet_service.cc:1467] Processing CreateTablet for tablet 207eb00544e74e1caa151902b5d30381 (DEFAULT_TABLE table=test-table [id=047be7fdd2614d1399c3f717584e17d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:59:25.214640 24383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 207eb00544e74e1caa151902b5d30381. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:59:25.215626 24461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:25.184170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:38391 (local address 127.0.0.1:60294)
0925 04:59:25.202994 (+ 18824us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:25.203012 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:25.203132 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:25.203874 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:25.203878 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:25.203903 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:59:25.204237 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:25.204248 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:25.206114 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:25.206119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:25.212710 (+ 6591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:25.212728 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:25.214882 (+ 2154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:25.214889 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:25.215061 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:25.215093 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 04:59:25.215183 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 04:59:25.215320 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18703,"thread_start_us":15789,"threads_started":1}
I20260925 04:59:25.215845 24465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:25.184468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60294 (local address 127.6.29.129:38391)
0925 04:59:25.197139 (+ 12671us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:25.197163 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:25.203146 (+ 5983us) server_negotiation.cc:407] Connection header received
0925 04:59:25.203280 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:25.203289 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:25.203553 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:25.203747 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:59:25.204441 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:25.205959 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:25.212949 (+ 6990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:25.214695 (+ 1746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:25.214841 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:25.215113 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:25.215270 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 04:59:25.215452 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12407,"thread_start_us":12711,"threads_started":1}
I20260925 04:59:25.219812 24122 tablet_service.cc:1467] Processing CreateTablet for tablet 207eb00544e74e1caa151902b5d30381 (DEFAULT_TABLE table=test-table [id=047be7fdd2614d1399c3f717584e17d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:59:25.221417 24122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 207eb00544e74e1caa151902b5d30381. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:59:25.228978 24466 tablet_bootstrap.cc:492] T 207eb00544e74e1caa151902b5d30381 P 671dcaacd498452a8dcfe825783fdbe4: Bootstrap starting.
I20260925 04:59:25.231106 24467 tablet_bootstrap.cc:492] T 207eb00544e74e1caa151902b5d30381 P c6b50966ef604bd38cfe92d83979564e: Bootstrap starting.
I20260925 04:59:25.232990 24466 tablet_bootstrap.cc:654] T 207eb00544e74e1caa151902b5d30381 P 671dcaacd498452a8dcfe825783fdbe4: Neither blocks nor log segments found. Creating new log.
I20260925 04:59:25.234369 24466 log.cc:824] T 207eb00544e74e1caa151902b5d30381 P 671dcaacd498452a8dcfe825783fdbe4: Log is configured to *not* fsync() on all Append() calls
I20260925 04:59:25.234977 24467 tablet_bootstrap.cc:654] T 207eb00544e74e1caa151902b5d30381 P c6b50966ef604bd38cfe92d83979564e: Neither blocks nor log segments found. Creating new log.
I20260925 04:59:25.236371 24467 log.cc:824] T 207eb00544e74e1caa151902b5d30381 P c6b50966ef604bd38cfe92d83979564e: Log is configured to *not* fsync() on all Append() calls
I20260925 04:59:25.237828 24469 tablet_bootstrap.cc:492] T 207eb00544e74e1caa151902b5d30381 P 7da9920def4049d0a3b22aa9c572fae3: Bootstrap starting.
I20260925 04:59:25.238170 24466 tablet_bootstrap.cc:492] T 207eb00544e74e1caa151902b5d30381 P 671dcaacd498452a8dcfe825783fdbe4: No bootstrap required, opened a new log
I20260925 04:59:25.238468 24466 ts_tablet_manager.cc:1397] T 207eb00544e74e1caa151902b5d30381 P 671dcaacd498452a8dcfe825783fdbe4: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260925 04:59:25.239854 24467 tablet_bootstrap.cc:492] T 207eb00544e74e1caa151902b5d30381 P c6b50966ef604bd38cfe92d83979564e: No bootstrap required, opened a new log
I20260925 04:59:25.240123 24467 ts_tablet_manager.cc:1397] T 207eb00544e74e1caa151902b5d30381 P c6b50966ef604bd38cfe92d83979564e: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260925 04:59:25.241592 24469 tablet_bootstrap.cc:654] T 207eb00544e74e1caa151902b5d30381 P 7da9920def4049d0a3b22aa9c572fae3: Neither blocks nor log segments found. Creating new log.
I20260925 04:59:25.242801 24469 log.cc:824] T 207eb00544e74e1caa151902b5d30381 P 7da9920def4049d0a3b22aa9c572fae3: Log is configured to *not* fsync() on all Append() calls
I20260925 04:59:25.246025 24469 tablet_bootstrap.cc:492] T 207eb00544e74e1caa151902b5d30381 P 7da9920def4049d0a3b22aa9c572fae3: No bootstrap required, opened a new log
I20260925 04:59:25.246315 24469 ts_tablet_manager.cc:1397] T 207eb00544e74e1caa151902b5d30381 P 7da9920def4049d0a3b22aa9c572fae3: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260925 04:59:25.247148 24466 raft_consensus.cc:348] T 207eb00544e74e1caa151902b5d30381 P 671dcaacd498452a8dcfe825783fdbe4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671dcaacd498452a8dcfe825783fdbe4" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 39399 } } peers { permanent_uuid: "c6b50966ef604bd38cfe92d83979564e" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43619 } } peers { permanent_uuid: "7da9920def4049d0a3b22aa9c572fae3" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 38391 } }
I20260925 04:59:25.247622 24466 raft_consensus.cc:374] T 207eb00544e74e1caa151902b5d30381 P 671dcaacd498452a8dcfe825783fdbe4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:59:25.247721 24466 raft_consensus.cc:729] T 207eb00544e74e1caa151902b5d30381 P 671dcaacd498452a8dcfe825783fdbe4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 671dcaacd498452a8dcfe825783fdbe4, State: Initialized, Role: FOLLOWER
I20260925 04:59:25.248526 24466 consensus_queue.cc:260] T 207eb00544e74e1caa151902b5d30381 P 671dcaacd498452a8dcfe825783fdbe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671dcaacd498452a8dcfe825783fdbe4" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 39399 } } peers { permanent_uuid: "c6b50966ef604bd38cfe92d83979564e" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43619 } } peers { permanent_uuid: "7da9920def4049d0a3b22aa9c572fae3" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 38391 } }
I20260925 04:59:25.249089 24467 raft_consensus.cc:348] T 207eb00544e74e1caa151902b5d30381 P c6b50966ef604bd38cfe92d83979564e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671dcaacd498452a8dcfe825783fdbe4" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 39399 } } peers { permanent_uuid: "c6b50966ef604bd38cfe92d83979564e" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43619 } } peers { permanent_uuid: "7da9920def4049d0a3b22aa9c572fae3" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 38391 } }
I20260925 04:59:25.249549 24467 raft_consensus.cc:374] T 207eb00544e74e1caa151902b5d30381 P c6b50966ef604bd38cfe92d83979564e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:59:25.249658 24467 raft_consensus.cc:729] T 207eb00544e74e1caa151902b5d30381 P c6b50966ef604bd38cfe92d83979564e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6b50966ef604bd38cfe92d83979564e, State: Initialized, Role: FOLLOWER
I20260925 04:59:25.249795 24318 heartbeater.cc:503] Master 127.6.29.190:35643 was elected leader, sending a full tablet report...
I20260925 04:59:25.250239 24467 consensus_queue.cc:260] T 207eb00544e74e1caa151902b5d30381 P c6b50966ef604bd38cfe92d83979564e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671dcaacd498452a8dcfe825783fdbe4" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 39399 } } peers { permanent_uuid: "c6b50966ef604bd38cfe92d83979564e" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43619 } } peers { permanent_uuid: "7da9920def4049d0a3b22aa9c572fae3" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 38391 } }
I20260925 04:59:25.251291 24448 heartbeater.cc:503] Master 127.6.29.190:35643 was elected leader, sending a full tablet report...
I20260925 04:59:25.252049 24467 ts_tablet_manager.cc:1428] T 207eb00544e74e1caa151902b5d30381 P c6b50966ef604bd38cfe92d83979564e: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260925 04:59:25.254801 24466 ts_tablet_manager.cc:1428] T 207eb00544e74e1caa151902b5d30381 P 671dcaacd498452a8dcfe825783fdbe4: Time spent starting tablet: real 0.016s user 0.012s sys 0.000s
I20260925 04:59:25.257238 24469 raft_consensus.cc:348] T 207eb00544e74e1caa151902b5d30381 P 7da9920def4049d0a3b22aa9c572fae3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671dcaacd498452a8dcfe825783fdbe4" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 39399 } } peers { permanent_uuid: "c6b50966ef604bd38cfe92d83979564e" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43619 } } peers { permanent_uuid: "7da9920def4049d0a3b22aa9c572fae3" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 38391 } }
I20260925 04:59:25.257612 24469 raft_consensus.cc:374] T 207eb00544e74e1caa151902b5d30381 P 7da9920def4049d0a3b22aa9c572fae3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:59:25.257728 24469 raft_consensus.cc:729] T 207eb00544e74e1caa151902b5d30381 P 7da9920def4049d0a3b22aa9c572fae3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7da9920def4049d0a3b22aa9c572fae3, State: Initialized, Role: FOLLOWER
I20260925 04:59:25.258339 24469 consensus_queue.cc:260] T 207eb00544e74e1caa151902b5d30381 P 7da9920def4049d0a3b22aa9c572fae3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671dcaacd498452a8dcfe825783fdbe4" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 39399 } } peers { permanent_uuid: "c6b50966ef604bd38cfe92d83979564e" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43619 } } peers { permanent_uuid: "7da9920def4049d0a3b22aa9c572fae3" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 38391 } }
I20260925 04:59:25.260057 24469 ts_tablet_manager.cc:1428] T 207eb00544e74e1caa151902b5d30381 P 7da9920def4049d0a3b22aa9c572fae3: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260925 04:59:25.280866 24473 raft_consensus.cc:482] T 207eb00544e74e1caa151902b5d30381 P c6b50966ef604bd38cfe92d83979564e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:59:25.281211 24473 raft_consensus.cc:504] T 207eb00544e74e1caa151902b5d30381 P c6b50966ef604bd38cfe92d83979564e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671dcaacd498452a8dcfe825783fdbe4" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 39399 } } peers { permanent_uuid: "c6b50966ef604bd38cfe92d83979564e" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43619 } } peers { permanent_uuid: "7da9920def4049d0a3b22aa9c572fae3" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 38391 } }
I20260925 04:59:25.282886 24473 leader_election.cc:288] T 207eb00544e74e1caa151902b5d30381 P c6b50966ef604bd38cfe92d83979564e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 671dcaacd498452a8dcfe825783fdbe4 (127.6.29.130:39399), 7da9920def4049d0a3b22aa9c572fae3 (127.6.29.129:38391)
I20260925 04:59:25.289870 24462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:25.283492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:33457 (local address 127.6.29.130:39399)
0925 04:59:25.283708 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:25.283724 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:25.283746 (+ 22us) server_negotiation.cc:407] Connection header received
0925 04:59:25.284001 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:25.284009 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:25.284107 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:25.284216 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:59:25.284766 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:25.285730 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:25.287728 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:25.289002 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:25.289117 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:25.289167 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:25.289526 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 04:59:25.289656 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 04:59:25.289996 24450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:25.283239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:39399 (local address 127.6.29.131:33457)
0925 04:59:25.283569 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:25.283595 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:25.283755 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:25.284266 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:25.284269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:25.284297 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:59:25.284530 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:25.284541 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:25.285860 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:25.285866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:25.287536 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:25.287549 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:25.289157 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:25.289163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:25.289361 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:25.289398 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 04:59:25.289507 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 04:59:25.289677 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":4}
I20260925 04:59:25.290889 24465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:25.283492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:38387 (local address 127.6.29.129:38391)
0925 04:59:25.283709 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:25.283714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:25.284412 (+ 698us) server_negotiation.cc:407] Connection header received
0925 04:59:25.284568 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:25.284573 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:25.284701 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:25.284772 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:59:25.285428 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:25.286385 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:25.288864 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:25.290078 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:25.290259 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:25.290349 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:25.290579 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 04:59:25.290720 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 04:59:25.291003 24475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:25.283298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:38391 (local address 127.6.29.131:38387)
0925 04:59:25.284296 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:25.284314 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:25.284438 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:25.284827 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:25.284834 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:25.284864 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:59:25.285259 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:25.285271 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:25.286545 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:25.286549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:25.288656 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:25.288672 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:25.290266 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:25.290275 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:25.290424 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:25.290446 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:59:25.290519 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 04:59:25.290643 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"mutex_wait_us":37,"thread_start_us":605,"threads_started":1}
I20260925 04:59:25.291148 24273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "207eb00544e74e1caa151902b5d30381" candidate_uuid: "c6b50966ef604bd38cfe92d83979564e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "671dcaacd498452a8dcfe825783fdbe4" is_pre_election: true
I20260925 04:59:25.291824 24273 raft_consensus.cc:2454] T 207eb00544e74e1caa151902b5d30381 P 671dcaacd498452a8dcfe825783fdbe4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6b50966ef604bd38cfe92d83979564e in term 0.
I20260925 04:59:25.292104 24142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "207eb00544e74e1caa151902b5d30381" candidate_uuid: "c6b50966ef604bd38cfe92d83979564e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da9920def4049d0a3b22aa9c572fae3" is_pre_election: true
I20260925 04:59:25.292502 24337 leader_election.cc:302] T 207eb00544e74e1caa151902b5d30381 P c6b50966ef604bd38cfe92d83979564e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 671dcaacd498452a8dcfe825783fdbe4, c6b50966ef604bd38cfe92d83979564e; no voters:
I20260925 04:59:25.292691 24142 raft_consensus.cc:2454] T 207eb00544e74e1caa151902b5d30381 P 7da9920def4049d0a3b22aa9c572fae3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6b50966ef604bd38cfe92d83979564e in term 0.
I20260925 04:59:25.292888 24473 raft_consensus.cc:2781] T 207eb00544e74e1caa151902b5d30381 P c6b50966ef604bd38cfe92d83979564e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:59:25.293027 24473 raft_consensus.cc:482] T 207eb00544e74e1caa151902b5d30381 P c6b50966ef604bd38cfe92d83979564e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:59:25.293135 24473 raft_consensus.cc:3037] T 207eb00544e74e1caa151902b5d30381 P c6b50966ef604bd38cfe92d83979564e [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:59:25.295346 24473 raft_consensus.cc:504] T 207eb00544e74e1caa151902b5d30381 P c6b50966ef604bd38cfe92d83979564e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671dcaacd498452a8dcfe825783fdbe4" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 39399 } } peers { permanent_uuid: "c6b50966ef604bd38cfe92d83979564e" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43619 } } peers { permanent_uuid: "7da9920def4049d0a3b22aa9c572fae3" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 38391 } }
I20260925 04:59:25.296056 24473 leader_election.cc:288] T 207eb00544e74e1caa151902b5d30381 P c6b50966ef604bd38cfe92d83979564e [CANDIDATE]: Term 1 election: Requested vote from peers 671dcaacd498452a8dcfe825783fdbe4 (127.6.29.130:39399), 7da9920def4049d0a3b22aa9c572fae3 (127.6.29.129:38391)
I20260925 04:59:25.296439 24273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "207eb00544e74e1caa151902b5d30381" candidate_uuid: "c6b50966ef604bd38cfe92d83979564e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "671dcaacd498452a8dcfe825783fdbe4"
I20260925 04:59:25.296592 24142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "207eb00544e74e1caa151902b5d30381" candidate_uuid: "c6b50966ef604bd38cfe92d83979564e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7da9920def4049d0a3b22aa9c572fae3"
I20260925 04:59:25.296643 24273 raft_consensus.cc:3037] T 207eb00544e74e1caa151902b5d30381 P 671dcaacd498452a8dcfe825783fdbe4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:59:25.296804 24142 raft_consensus.cc:3037] T 207eb00544e74e1caa151902b5d30381 P 7da9920def4049d0a3b22aa9c572fae3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:59:25.299170 24273 raft_consensus.cc:2454] T 207eb00544e74e1caa151902b5d30381 P 671dcaacd498452a8dcfe825783fdbe4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6b50966ef604bd38cfe92d83979564e in term 1.
I20260925 04:59:25.299261 24142 raft_consensus.cc:2454] T 207eb00544e74e1caa151902b5d30381 P 7da9920def4049d0a3b22aa9c572fae3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6b50966ef604bd38cfe92d83979564e in term 1.
I20260925 04:59:25.299557 24337 leader_election.cc:302] T 207eb00544e74e1caa151902b5d30381 P c6b50966ef604bd38cfe92d83979564e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 671dcaacd498452a8dcfe825783fdbe4, c6b50966ef604bd38cfe92d83979564e; no voters:
I20260925 04:59:25.299857 24473 raft_consensus.cc:2781] T 207eb00544e74e1caa151902b5d30381 P c6b50966ef604bd38cfe92d83979564e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:59:25.300328 24473 raft_consensus.cc:686] T 207eb00544e74e1caa151902b5d30381 P c6b50966ef604bd38cfe92d83979564e [term 1 LEADER]: Becoming Leader. State: Replica: c6b50966ef604bd38cfe92d83979564e, State: Running, Role: LEADER
I20260925 04:59:25.300851 24473 consensus_queue.cc:237] T 207eb00544e74e1caa151902b5d30381 P c6b50966ef604bd38cfe92d83979564e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671dcaacd498452a8dcfe825783fdbe4" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 39399 } } peers { permanent_uuid: "c6b50966ef604bd38cfe92d83979564e" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43619 } } peers { permanent_uuid: "7da9920def4049d0a3b22aa9c572fae3" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 38391 } }
I20260925 04:59:25.305349 24001 catalog_manager.cc:5495] T 207eb00544e74e1caa151902b5d30381 P c6b50966ef604bd38cfe92d83979564e reported cstate change: term changed from 0 to 1, leader changed from <none> to c6b50966ef604bd38cfe92d83979564e (127.6.29.131). New cstate: current_term: 1 leader_uuid: "c6b50966ef604bd38cfe92d83979564e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671dcaacd498452a8dcfe825783fdbe4" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 39399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6b50966ef604bd38cfe92d83979564e" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7da9920def4049d0a3b22aa9c572fae3" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 38391 } health_report { overall_health: UNKNOWN } } }
I20260925 04:59:25.333379 24463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:25.328003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50766 (local address 127.6.29.131:43619)
0925 04:59:25.328214 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:25.328220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:25.328238 (+ 18us) server_negotiation.cc:407] Connection header received
0925 04:59:25.328452 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:25.328455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:25.328535 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:25.328716 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:59:25.329066 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:25.329809 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:25.331221 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:25.331631 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:25.331770 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:25.332284 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:59:25.332791 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:59:25.332947 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:25.333048 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 04:59:25.333124 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 04:59:25.341641 24142 raft_consensus.cc:1261] T 207eb00544e74e1caa151902b5d30381 P 7da9920def4049d0a3b22aa9c572fae3 [term 1 FOLLOWER]: Refusing update from remote peer c6b50966ef604bd38cfe92d83979564e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:59:25.341639 24273 raft_consensus.cc:1261] T 207eb00544e74e1caa151902b5d30381 P 671dcaacd498452a8dcfe825783fdbe4 [term 1 FOLLOWER]: Refusing update from remote peer c6b50966ef604bd38cfe92d83979564e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:59:25.342562 24476 consensus_queue.cc:1035] T 207eb00544e74e1caa151902b5d30381 P c6b50966ef604bd38cfe92d83979564e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7da9920def4049d0a3b22aa9c572fae3" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 38391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:59:25.342952 24473 consensus_queue.cc:1035] T 207eb00544e74e1caa151902b5d30381 P c6b50966ef604bd38cfe92d83979564e [LEADER]: Connected to new peer: Peer: permanent_uuid: "671dcaacd498452a8dcfe825783fdbe4" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 39399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:59:25.354514 24479 mvcc.cc:205] Tried to move back new op lower bound from 7333119448428662784 to 7333119448276410368. Current Snapshot: MvccSnapshot[applied={T|T < 7333119448428662784}]
I20260925 04:59:25.359444 24480 mvcc.cc:205] Tried to move back new op lower bound from 7333119448428662784 to 7333119448276410368. Current Snapshot: MvccSnapshot[applied={T|T < 7333119448428662784 or (T in {7333119448428662784})}]
I20260925 04:59:25.368722 24000 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:59:25.369010 24000 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50660:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:59:25.372742 24000 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 18948d6de4014a7a8eec5ff6122369ec: Sending DeleteTablet for 3 replicas of tablet 207eb00544e74e1caa151902b5d30381
I20260925 04:59:25.373945 24253 tablet_service.cc:1514] Processing DeleteTablet for tablet 207eb00544e74e1caa151902b5d30381 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:59:25 UTC) from {username='slave'} at 127.0.0.1:58604
I20260925 04:59:25.374714 24383 tablet_service.cc:1514] Processing DeleteTablet for tablet 207eb00544e74e1caa151902b5d30381 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:59:25 UTC) from {username='slave'} at 127.0.0.1:50752
I20260925 04:59:25.374835 24488 tablet_replica.cc:330] T 207eb00544e74e1caa151902b5d30381 P 671dcaacd498452a8dcfe825783fdbe4: stopping tablet replica
I20260925 04:59:25.375262 24488 raft_consensus.cc:2229] T 207eb00544e74e1caa151902b5d30381 P 671dcaacd498452a8dcfe825783fdbe4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:59:25.375700 24489 tablet_replica.cc:330] T 207eb00544e74e1caa151902b5d30381 P c6b50966ef604bd38cfe92d83979564e: stopping tablet replica
I20260925 04:59:25.375833 24488 raft_consensus.cc:2258] T 207eb00544e74e1caa151902b5d30381 P 671dcaacd498452a8dcfe825783fdbe4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:59:25.376166 24122 tablet_service.cc:1514] Processing DeleteTablet for tablet 207eb00544e74e1caa151902b5d30381 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:59:25 UTC) from {username='slave'} at 127.0.0.1:60294
I20260925 04:59:25.376246 24489 raft_consensus.cc:2229] T 207eb00544e74e1caa151902b5d30381 P c6b50966ef604bd38cfe92d83979564e [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:59:25.377045 24490 tablet_replica.cc:330] T 207eb00544e74e1caa151902b5d30381 P 7da9920def4049d0a3b22aa9c572fae3: stopping tablet replica
I20260925 04:59:25.376991 24489 raft_consensus.cc:2258] T 207eb00544e74e1caa151902b5d30381 P c6b50966ef604bd38cfe92d83979564e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:59:25.377449 24490 raft_consensus.cc:2229] T 207eb00544e74e1caa151902b5d30381 P 7da9920def4049d0a3b22aa9c572fae3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:59:25.377948 24490 raft_consensus.cc:2258] T 207eb00544e74e1caa151902b5d30381 P 7da9920def4049d0a3b22aa9c572fae3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:59:25.378017 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 24060
I20260925 04:59:25.381150 24490 ts_tablet_manager.cc:1905] T 207eb00544e74e1caa151902b5d30381 P 7da9920def4049d0a3b22aa9c572fae3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:59:25.382963 24488 ts_tablet_manager.cc:1905] T 207eb00544e74e1caa151902b5d30381 P 671dcaacd498452a8dcfe825783fdbe4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:59:25.387796 24465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:25.379215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60298 (local address 127.6.29.129:38391)
0925 04:59:25.379569 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:25.379576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:25.379595 (+ 19us) server_negotiation.cc:407] Connection header received
0925 04:59:25.379682 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:25.379688 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:25.379823 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:25.379930 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:25.380441 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:25.381519 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:25.383679 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:25.384346 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:25.384437 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:25.385991 (+ 1554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:25.386002 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:25.386018 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:59:25.386792 (+ 774us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:25.386849 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:59:25.386878 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:25.386966 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:25.387281 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 04:59:25.387393 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214}
I20260925 04:59:25.388350 24490 ts_tablet_manager.cc:1918] T 207eb00544e74e1caa151902b5d30381 P 7da9920def4049d0a3b22aa9c572fae3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:59:25.388535 24490 log.cc:1196] T 207eb00544e74e1caa151902b5d30381 P 7da9920def4049d0a3b22aa9c572fae3: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-0/wal/wals/207eb00544e74e1caa151902b5d30381
I20260925 04:59:25.389140 24490 ts_tablet_manager.cc:1939] T 207eb00544e74e1caa151902b5d30381 P 7da9920def4049d0a3b22aa9c572fae3: Deleting consensus metadata
I20260925 04:59:25.389210 24489 ts_tablet_manager.cc:1905] T 207eb00544e74e1caa151902b5d30381 P c6b50966ef604bd38cfe92d83979564e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:59:25.389514 24182 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:59:25.397826 24488 ts_tablet_manager.cc:1918] T 207eb00544e74e1caa151902b5d30381 P 671dcaacd498452a8dcfe825783fdbe4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:59:25.398015 24488 log.cc:1196] T 207eb00544e74e1caa151902b5d30381 P 671dcaacd498452a8dcfe825783fdbe4: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-1/wal/wals/207eb00544e74e1caa151902b5d30381
I20260925 04:59:25.398597 24488 ts_tablet_manager.cc:1939] T 207eb00544e74e1caa151902b5d30381 P 671dcaacd498452a8dcfe825783fdbe4: Deleting consensus metadata
I20260925 04:59:25.398959 24489 ts_tablet_manager.cc:1918] T 207eb00544e74e1caa151902b5d30381 P c6b50966ef604bd38cfe92d83979564e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:59:25.399159 24489 log.cc:1196] T 207eb00544e74e1caa151902b5d30381 P c6b50966ef604bd38cfe92d83979564e: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790312053666503-6262-0/minicluster-data/ts-2/wal/wals/207eb00544e74e1caa151902b5d30381
I20260925 04:59:25.399700 24489 ts_tablet_manager.cc:1939] T 207eb00544e74e1caa151902b5d30381 P c6b50966ef604bd38cfe92d83979564e: Deleting consensus metadata
I20260925 04:59:25.400852 23986 catalog_manager.cc:4841] TS 671dcaacd498452a8dcfe825783fdbe4 (127.6.29.130:39399): tablet 207eb00544e74e1caa151902b5d30381 (table test-table [id=047be7fdd2614d1399c3f717584e17d0]) successfully deleted
I20260925 04:59:25.401413 23985 catalog_manager.cc:4841] TS c6b50966ef604bd38cfe92d83979564e (127.6.29.131:43619): tablet 207eb00544e74e1caa151902b5d30381 (table test-table [id=047be7fdd2614d1399c3f717584e17d0]) successfully deleted
I20260925 04:59:25.513250 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 24060
I20260925 04:59:25.513381 23988 catalog_manager.cc:4841] TS 7da9920def4049d0a3b22aa9c572fae3 (127.6.29.129:38391): tablet 207eb00544e74e1caa151902b5d30381 (table test-table [id=047be7fdd2614d1399c3f717584e17d0]) successfully deleted
I20260925 04:59:25.531635 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 24191
I20260925 04:59:25.539587 24462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:25.532688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58616 (local address 127.6.29.130:39399)
0925 04:59:25.532865 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:25.532869 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:25.532889 (+ 20us) server_negotiation.cc:407] Connection header received
0925 04:59:25.532983 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:25.532987 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:25.533078 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:25.533169 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:25.533610 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:25.534447 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:25.536390 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:25.536811 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:25.536915 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:25.538101 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:25.538110 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:25.538167 (+ 57us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:59:25.538770 (+ 603us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:25.538821 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:59:25.538842 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:25.538935 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:25.539112 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 04:59:25.539222 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 04:59:25.540544 24313 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:59:25.630335 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 24191
I20260925 04:59:25.648674 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 24321
I20260925 04:59:25.657485 24463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:25.649514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50780 (local address 127.6.29.131:43619)
0925 04:59:25.649942 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:25.649950 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:25.649964 (+ 14us) server_negotiation.cc:407] Connection header received
0925 04:59:25.650016 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:25.650019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:25.650133 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:25.650210 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:25.650731 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:25.651539 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:25.653565 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:25.654217 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:25.654334 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:25.655676 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:25.655686 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:25.655701 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:59:25.656567 (+ 866us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:25.656619 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:59:25.656640 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:25.656744 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:25.657018 (+ 274us) server_negotiation.cc:299] Negotiation successful
0925 04:59:25.657137 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309}
I20260925 04:59:25.658226 24443 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:59:25.780140 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 24321
I20260925 04:59:25.800377 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 23970
I20260925 04:59:25.801489 24031 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:59:25.912499 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 23970
2026-09-25T04:59:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2807 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:59:26 dist-test-slave-xzjr krb5kdc[24498](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:59:26 dist-test-slave-xzjr krb5kdc[24498](info): set up 2 sockets
Sep 25 04:59:26 dist-test-slave-xzjr krb5kdc[24498](info): commencing operation
krb5kdc: starting...
W20260925 04:59:27.028481 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 04:59:27 dist-test-slave-xzjr krb5kdc[24498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:59:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:59:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.29.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
I20260925 04:59:27.307166 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:33089
--webserver_interface=127.6.29.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:40869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:33089
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:59:27.641573 24511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:59:27.641927 24511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:59:27.642103 24511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:59:27.649633 24511 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:59:27.649771 24511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:59:27.649870 24511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:59:27.649955 24511 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:59:27.650022 24511 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:59:27.660336 24511 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:40869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:33089
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:33089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:59:27.661831 24511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:59:27.663405 24511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:59:27.675839 24516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:27.675786 24519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:27.675782 24517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:59:27.676357 24511 server_base.cc:1034] running on GCE node
I20260925 04:59:27.677320 24511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:59:27.678897 24511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:59:27.680163 24511 hybrid_clock.cc:648] HybridClock initialized: now 1790312367680065 us; error 48 us; skew 500 ppm
I20260925 04:59:27.680547 24511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:59:27 dist-test-slave-xzjr krb5kdc[24498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312367, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:59:27.683887 24511 init.cc:377] Logged in from keytab as kudu/127.6.29.190@KRBTEST.COM (short username kudu)
I20260925 04:59:27.685508 24511 webserver.cc:460] Webserver started at http://127.6.29.190:32839/ using document root <none> and password file <none>
I20260925 04:59:27.686230 24511 fs_manager.cc:359] Metadata directory not provided
I20260925 04:59:27.686372 24511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:59:27.686686 24511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:59:27.689859 24511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "f2481b34bd954464a93d56cd9cc99bbc"
format_stamp: "Formatted at 2026-09-25 04:59:27 on dist-test-slave-xzjr"
server_key: "6cb46caf4da05c18862a678ccdd64d58"
server_key_iv: "e4ad2d4f36ce6ffed953c69fc4982671"
server_key_version: "encryptionkey@0"
I20260925 04:59:27.690739 24511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "f2481b34bd954464a93d56cd9cc99bbc"
format_stamp: "Formatted at 2026-09-25 04:59:27 on dist-test-slave-xzjr"
server_key: "6cb46caf4da05c18862a678ccdd64d58"
server_key_iv: "e4ad2d4f36ce6ffed953c69fc4982671"
server_key_version: "encryptionkey@0"
I20260925 04:59:27.696242 24511 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 04:59:27.699963 24526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:27.701431 24511 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:59:27.701612 24511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "f2481b34bd954464a93d56cd9cc99bbc"
format_stamp: "Formatted at 2026-09-25 04:59:27 on dist-test-slave-xzjr"
server_key: "6cb46caf4da05c18862a678ccdd64d58"
server_key_iv: "e4ad2d4f36ce6ffed953c69fc4982671"
server_key_version: "encryptionkey@0"
I20260925 04:59:27.701826 24511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:59:27.722685 24511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:59:27.727346 24511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:59:27.727664 24511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:59:27.746767 24511 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:33089
I20260925 04:59:27.746801 24577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:33089 every 8 connection(s)
I20260925 04:59:27.748262 24511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:59:27.751539 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 24511
I20260925 04:59:27.751747 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 04:59:27.752347 6262 external_mini_cluster.cc:1428] Setting key 469e4685678a7632ac004da6e7fc6772
I20260925 04:59:27.752503 24578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:59:27.767037 24578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2481b34bd954464a93d56cd9cc99bbc: Bootstrap starting.
Sep 25 04:59:27 dist-test-slave-xzjr krb5kdc[24498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:59:27.770793 24578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2481b34bd954464a93d56cd9cc99bbc: Neither blocks nor log segments found. Creating new log.
I20260925 04:59:27.772027 24578 log.cc:824] T 00000000000000000000000000000000 P f2481b34bd954464a93d56cd9cc99bbc: Log is configured to *not* fsync() on all Append() calls
I20260925 04:59:27.775357 24578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2481b34bd954464a93d56cd9cc99bbc: No bootstrap required, opened a new log
I20260925 04:59:27.777151 24581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:27.754232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49556 (local address 127.6.29.190:33089)
0925 04:59:27.755020 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:27.755035 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:27.755086 (+ 51us) server_negotiation.cc:407] Connection header received
0925 04:59:27.757607 (+ 2521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:27.757637 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:27.758205 (+ 568us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:27.758675 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:27.760090 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:27.762111 (+ 2021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:27.763480 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:27.764622 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:27.764818 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:27.769338 (+ 4520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:27.769345 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:27.769359 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:59:27.769407 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:27.772902 (+ 3495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:27.773790 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:27.773797 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:27.773803 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:27.773961 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:27.774402 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:27.774406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:27.774408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:27.774917 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:27.775142 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:27.775666 (+ 524us) server_negotiation.cc:299] Negotiation successful
0925 04:59:27.776136 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":329,"threads_started":1}
I20260925 04:59:27.783751 24578 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2481b34bd954464a93d56cd9cc99bbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2481b34bd954464a93d56cd9cc99bbc" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33089 } }
I20260925 04:59:27.784210 24578 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2481b34bd954464a93d56cd9cc99bbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:59:27.784313 24578 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2481b34bd954464a93d56cd9cc99bbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2481b34bd954464a93d56cd9cc99bbc, State: Initialized, Role: FOLLOWER
I20260925 04:59:27.784909 24578 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2481b34bd954464a93d56cd9cc99bbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2481b34bd954464a93d56cd9cc99bbc" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33089 } }
I20260925 04:59:27.785175 24578 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2481b34bd954464a93d56cd9cc99bbc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:59:27.785321 24578 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2481b34bd954464a93d56cd9cc99bbc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:59:27.785516 24578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2481b34bd954464a93d56cd9cc99bbc [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:59:27.787832 24578 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2481b34bd954464a93d56cd9cc99bbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2481b34bd954464a93d56cd9cc99bbc" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33089 } }
I20260925 04:59:27.788350 24578 leader_election.cc:302] T 00000000000000000000000000000000 P f2481b34bd954464a93d56cd9cc99bbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2481b34bd954464a93d56cd9cc99bbc; no voters:
I20260925 04:59:27.789110 24578 leader_election.cc:288] T 00000000000000000000000000000000 P f2481b34bd954464a93d56cd9cc99bbc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:59:27.789415 24583 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2481b34bd954464a93d56cd9cc99bbc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:59:27.790237 24583 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2481b34bd954464a93d56cd9cc99bbc [term 1 LEADER]: Becoming Leader. State: Replica: f2481b34bd954464a93d56cd9cc99bbc, State: Running, Role: LEADER
I20260925 04:59:27.790787 24583 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2481b34bd954464a93d56cd9cc99bbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2481b34bd954464a93d56cd9cc99bbc" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33089 } }
I20260925 04:59:27.791384 24578 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2481b34bd954464a93d56cd9cc99bbc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:59:27.793578 24584 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2481b34bd954464a93d56cd9cc99bbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2481b34bd954464a93d56cd9cc99bbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2481b34bd954464a93d56cd9cc99bbc" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33089 } } }
I20260925 04:59:27.793985 24584 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2481b34bd954464a93d56cd9cc99bbc [sys.catalog]: This master's current role is: LEADER
I20260925 04:59:27.794414 24585 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2481b34bd954464a93d56cd9cc99bbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2481b34bd954464a93d56cd9cc99bbc. Latest consensus state: current_term: 1 leader_uuid: "f2481b34bd954464a93d56cd9cc99bbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2481b34bd954464a93d56cd9cc99bbc" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33089 } } }
I20260925 04:59:27.794734 24585 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2481b34bd954464a93d56cd9cc99bbc [sys.catalog]: This master's current role is: LEADER
I20260925 04:59:27.794921 24590 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:59:27.802479 24590 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:59:27.811271 24590 catalog_manager.cc:1347] Generated new cluster ID: 82fad56d8c99411ab0f321253a455409
I20260925 04:59:27.811466 24590 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:59:27.833122 24590 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:59:27.834590 24590 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:59:27.840914 24590 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2481b34bd954464a93d56cd9cc99bbc: Generated new TSK 0
I20260925 04:59:27.842038 24590 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.29.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab.
Entry for principal kudu/127.6.29.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab.
I20260925 04:59:27.901264 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.129:0
--local_ip_for_outbound_sockets=127.6.29.129
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:33089
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:40869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:59:28.240324 24604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:59:28.240607 24604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:59:28.240701 24604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:59:28.247997 24604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:59:28.248117 24604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:59:28.248232 24604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.129
I20260925 04:59:28.258381 24604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:40869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:33089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.129
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:59:28.259699 24604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:59:28.261088 24604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:59:28.274065 24609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:28.274124 24612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:28.274147 24610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:59:28.275039 24604 server_base.cc:1034] running on GCE node
I20260925 04:59:28.275650 24604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:59:28.276641 24604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:59:28.277863 24604 hybrid_clock.cc:648] HybridClock initialized: now 1790312368277823 us; error 36 us; skew 500 ppm
I20260925 04:59:28.278188 24604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:59:28 dist-test-slave-xzjr krb5kdc[24498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312368, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:59:28.281450 24604 init.cc:377] Logged in from keytab as kudu/127.6.29.129@KRBTEST.COM (short username kudu)
I20260925 04:59:28.282948 24604 webserver.cc:460] Webserver started at http://127.6.29.129:44781/ using document root <none> and password file <none>
I20260925 04:59:28.283605 24604 fs_manager.cc:359] Metadata directory not provided
I20260925 04:59:28.283706 24604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:59:28.284045 24604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:59:28.286942 24604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-0/data/instance:
uuid: "9c63675290894ba187f03360b57ad509"
format_stamp: "Formatted at 2026-09-25 04:59:28 on dist-test-slave-xzjr"
server_key: "8682da1ed4d75409c826c0816a1f1dd2"
server_key_iv: "cb428831a6d0d2081e40b1ef7b20ba19"
server_key_version: "encryptionkey@0"
I20260925 04:59:28.287668 24604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance:
uuid: "9c63675290894ba187f03360b57ad509"
format_stamp: "Formatted at 2026-09-25 04:59:28 on dist-test-slave-xzjr"
server_key: "8682da1ed4d75409c826c0816a1f1dd2"
server_key_iv: "cb428831a6d0d2081e40b1ef7b20ba19"
server_key_version: "encryptionkey@0"
I20260925 04:59:28.292968 24604 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 04:59:28.296404 24619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:28.297886 24604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:59:28.298070 24604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-0/wal
uuid: "9c63675290894ba187f03360b57ad509"
format_stamp: "Formatted at 2026-09-25 04:59:28 on dist-test-slave-xzjr"
server_key: "8682da1ed4d75409c826c0816a1f1dd2"
server_key_iv: "cb428831a6d0d2081e40b1ef7b20ba19"
server_key_version: "encryptionkey@0"
I20260925 04:59:28.298280 24604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:59:28.321952 24604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:59:28.326463 24604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:59:28.326778 24604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:59:28.327828 24604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:59:28.329247 24604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:59:28.329375 24604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:28.329520 24604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:59:28.329592 24604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:28.361902 24604 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.129:34005
I20260925 04:59:28.361986 24731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.129:34005 every 8 connection(s)
I20260925 04:59:28.363231 24604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
I20260925 04:59:28.366160 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 24604
I20260925 04:59:28.366355 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance
I20260925 04:59:28.366791 6262 external_mini_cluster.cc:1428] Setting key aca8f034fefd7e23e20ceaab403537f8
Sep 25 04:59:28 dist-test-slave-xzjr krb5kdc[24498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312368, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:59:28.384194 24734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:28.365879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:42331 (local address 127.6.29.190:33089)
0925 04:59:28.366419 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:28.366425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:28.367527 (+ 1102us) server_negotiation.cc:407] Connection header received
0925 04:59:28.369424 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:28.369432 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:28.369518 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:28.369626 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:28.371131 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:28.372291 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:28.373780 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:28.374519 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:28.374662 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:28.378273 (+ 3611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:28.378278 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:28.378281 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:59:28.378330 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:28.380955 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:28.381884 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:28.381888 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:28.381890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:28.382005 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:28.382471 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:28.382474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:28.382476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:28.382727 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:28.382835 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:28.383712 (+ 877us) server_negotiation.cc:299] Negotiation successful
0925 04:59:28.383935 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":253,"threads_started":1}
I20260925 04:59:28.385735 24735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:28.366461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:33089 (local address 127.6.29.129:42331)
0925 04:59:28.367301 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:28.367346 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:28.369148 (+ 1802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:28.369849 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:28.369857 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:28.370372 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:59:28.370951 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:28.370969 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:28.372417 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:28.372420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:28.373633 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:28.373645 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:28.374732 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:28.374737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:28.374929 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:28.375642 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:59:28.375665 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:59:28.378068 (+ 2403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:59:28.381086 (+ 3018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:59:28.381097 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:59:28.381116 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:59:28.381763 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:59:28.382104 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:59:28.382112 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:59:28.382118 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:59:28.382366 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:59:28.382861 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:59:28.382872 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:59:28.383315 (+ 443us) client_negotiation.cc:769] Sending connection context
0925 04:59:28.383625 (+ 310us) client_negotiation.cc:240] Negotiation successful
0925 04:59:28.384252 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":338,"threads_started":1}
I20260925 04:59:28.387352 24732 heartbeater.cc:347] Connected to a master server at 127.6.29.190:33089
I20260925 04:59:28.387673 24732 heartbeater.cc:464] Registering TS with master...
I20260925 04:59:28.388471 24732 heartbeater.cc:511] Master 127.6.29.190:33089 requested a full tablet report, sending...
I20260925 04:59:28.390600 24543 ts_manager.cc:194] Registered new tserver with Master: 9c63675290894ba187f03360b57ad509 (127.6.29.129:34005)
WARNING: no policy specified for kudu/127.6.29.130@KRBTEST.COM; defaulting to no policy
I20260925 04:59:28.392181 24543 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.129@KRBTEST.COM'} at 127.6.29.129:42331
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab.
Entry for principal kudu/127.6.29.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab.
I20260925 04:59:28.423497 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.130:0
--local_ip_for_outbound_sockets=127.6.29.130
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:33089
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:40869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:59:28.756639 24738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:59:28.756971 24738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:59:28.757149 24738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:59:28.765214 24738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:59:28.765398 24738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:59:28.765591 24738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.130
I20260925 04:59:28.775561 24738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:40869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:33089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.130
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:59:28.777338 24738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:59:28.778676 24738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:59:28.792354 24746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:28.792426 24744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:28.792378 24743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:59:28.793200 24738 server_base.cc:1034] running on GCE node
I20260925 04:59:28.793840 24738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:59:28.794845 24738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:59:28.796068 24738 hybrid_clock.cc:648] HybridClock initialized: now 1790312368795993 us; error 69 us; skew 500 ppm
I20260925 04:59:28.796396 24738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:59:28 dist-test-slave-xzjr krb5kdc[24498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312368, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:59:28.804131 24738 init.cc:377] Logged in from keytab as kudu/127.6.29.130@KRBTEST.COM (short username kudu)
I20260925 04:59:28.805727 24738 webserver.cc:460] Webserver started at http://127.6.29.130:46575/ using document root <none> and password file <none>
I20260925 04:59:28.806432 24738 fs_manager.cc:359] Metadata directory not provided
I20260925 04:59:28.806540 24738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:59:28.806876 24738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:59:28.809867 24738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-1/data/instance:
uuid: "a5dca8adef804dc3a5961f72846ac507"
format_stamp: "Formatted at 2026-09-25 04:59:28 on dist-test-slave-xzjr"
server_key: "d91c2e8d1a26d4886e98b60b81d00214"
server_key_iv: "56dd28eb2d4fd33ff302ca170e56dc84"
server_key_version: "encryptionkey@0"
I20260925 04:59:28.810650 24738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance:
uuid: "a5dca8adef804dc3a5961f72846ac507"
format_stamp: "Formatted at 2026-09-25 04:59:28 on dist-test-slave-xzjr"
server_key: "d91c2e8d1a26d4886e98b60b81d00214"
server_key_iv: "56dd28eb2d4fd33ff302ca170e56dc84"
server_key_version: "encryptionkey@0"
I20260925 04:59:28.816032 24738 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 04:59:28.819885 24753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:28.821326 24738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:59:28.821511 24738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-1/wal
uuid: "a5dca8adef804dc3a5961f72846ac507"
format_stamp: "Formatted at 2026-09-25 04:59:28 on dist-test-slave-xzjr"
server_key: "d91c2e8d1a26d4886e98b60b81d00214"
server_key_iv: "56dd28eb2d4fd33ff302ca170e56dc84"
server_key_version: "encryptionkey@0"
I20260925 04:59:28.821731 24738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:59:28.843364 24738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:59:28.847723 24738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:59:28.848075 24738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:59:28.849112 24738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:59:28.850548 24738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:59:28.850626 24738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:28.850786 24738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:59:28.850837 24738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:28.883616 24738 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.130:38157
I20260925 04:59:28.883662 24865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.130:38157 every 8 connection(s)
I20260925 04:59:28.884997 24738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
I20260925 04:59:28.888626 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 24738
I20260925 04:59:28.888880 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance
I20260925 04:59:28.889485 6262 external_mini_cluster.cc:1428] Setting key f33604a7300cfea244b29c21abfa283e
Sep 25 04:59:28 dist-test-slave-xzjr krb5kdc[24498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312368, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:59:28.907980 24868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:28.887569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:57141 (local address 127.6.29.190:33089)
0925 04:59:28.888037 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:28.888042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:28.889344 (+ 1302us) server_negotiation.cc:407] Connection header received
0925 04:59:28.891732 (+ 2388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:28.891738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:28.891867 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:28.892002 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:28.893575 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:28.894326 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:28.895993 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:28.896519 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:28.896670 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:28.900389 (+ 3719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:28.900399 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:28.900407 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:59:28.900456 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:28.904190 (+ 3734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:28.904999 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:28.905007 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:28.905013 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:28.905183 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:28.905780 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:28.905787 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:28.905794 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:28.906183 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:28.906328 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:28.907164 (+ 836us) server_negotiation.cc:299] Negotiation successful
0925 04:59:28.907428 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":259,"threads_started":1}
I20260925 04:59:28.908670 24869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:28.888207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:33089 (local address 127.6.29.130:57141)
0925 04:59:28.889094 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:28.889208 (+ 114us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:28.891457 (+ 2249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:28.892261 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:28.892269 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:28.892805 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:59:28.893404 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:28.893422 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:28.894500 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:28.894509 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:28.895859 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:28.895871 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:28.896766 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:28.896773 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:28.897010 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:28.897772 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:59:28.897796 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:59:28.900115 (+ 2319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:59:28.904367 (+ 4252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:59:28.904375 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:59:28.904390 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:59:28.904857 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:59:28.905413 (+ 556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:59:28.905416 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:59:28.905419 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:59:28.905614 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:59:28.906333 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:59:28.906343 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:59:28.906708 (+ 365us) client_negotiation.cc:769] Sending connection context
0925 04:59:28.907004 (+ 296us) client_negotiation.cc:240] Negotiation successful
0925 04:59:28.907419 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":332,"threads_started":1}
I20260925 04:59:28.910235 24866 heartbeater.cc:347] Connected to a master server at 127.6.29.190:33089
I20260925 04:59:28.910600 24866 heartbeater.cc:464] Registering TS with master...
I20260925 04:59:28.911511 24866 heartbeater.cc:511] Master 127.6.29.190:33089 requested a full tablet report, sending...
I20260925 04:59:28.913260 24543 ts_manager.cc:194] Registered new tserver with Master: a5dca8adef804dc3a5961f72846ac507 (127.6.29.130:38157)
I20260925 04:59:28.914104 24543 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.130@KRBTEST.COM'} at 127.6.29.130:57141
WARNING: no policy specified for kudu/127.6.29.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab.
Entry for principal kudu/127.6.29.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab.
I20260925 04:59:28.948945 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.131:0
--local_ip_for_outbound_sockets=127.6.29.131
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:33089
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:40869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:59:29.285259 24872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:59:29.285550 24872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:59:29.285701 24872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:59:29.293231 24872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:59:29.293355 24872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:59:29.293540 24872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.131
I20260925 04:59:29.304126 24872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:40869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:33089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.131
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:59:29.305521 24872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:59:29.306910 24872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:59:29.320480 24880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:29.320529 24878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:29.320446 24877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:59:29.321368 24872 server_base.cc:1034] running on GCE node
I20260925 04:59:29.321988 24872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:59:29.322971 24872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:59:29.324199 24872 hybrid_clock.cc:648] HybridClock initialized: now 1790312369324154 us; error 44 us; skew 500 ppm
I20260925 04:59:29.324517 24872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:59:29 dist-test-slave-xzjr krb5kdc[24498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312369, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:59:29.327714 24872 init.cc:377] Logged in from keytab as kudu/127.6.29.131@KRBTEST.COM (short username kudu)
I20260925 04:59:29.329351 24872 webserver.cc:460] Webserver started at http://127.6.29.131:33505/ using document root <none> and password file <none>
I20260925 04:59:29.330042 24872 fs_manager.cc:359] Metadata directory not provided
I20260925 04:59:29.330189 24872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:59:29.330492 24872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:59:29.333464 24872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-2/data/instance:
uuid: "d188c59aabdb4b218a131656b843cf79"
format_stamp: "Formatted at 2026-09-25 04:59:29 on dist-test-slave-xzjr"
server_key: "692a26e646f9f8ba986a51987d0bbddc"
server_key_iv: "2b88ab493401da3a98a8aa342c4703c0"
server_key_version: "encryptionkey@0"
I20260925 04:59:29.334244 24872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance:
uuid: "d188c59aabdb4b218a131656b843cf79"
format_stamp: "Formatted at 2026-09-25 04:59:29 on dist-test-slave-xzjr"
server_key: "692a26e646f9f8ba986a51987d0bbddc"
server_key_iv: "2b88ab493401da3a98a8aa342c4703c0"
server_key_version: "encryptionkey@0"
I20260925 04:59:29.339404 24872 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 04:59:29.342868 24887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:29.344238 24872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:59:29.344404 24872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-2/wal
uuid: "d188c59aabdb4b218a131656b843cf79"
format_stamp: "Formatted at 2026-09-25 04:59:29 on dist-test-slave-xzjr"
server_key: "692a26e646f9f8ba986a51987d0bbddc"
server_key_iv: "2b88ab493401da3a98a8aa342c4703c0"
server_key_version: "encryptionkey@0"
I20260925 04:59:29.344605 24872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:59:29.395344 24732 heartbeater.cc:503] Master 127.6.29.190:33089 was elected leader, sending a full tablet report...
I20260925 04:59:29.396971 24872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:59:29.401813 24872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:59:29.402094 24872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:59:29.403199 24872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:59:29.404572 24872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:59:29.404652 24872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:29.404747 24872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:59:29.404790 24872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:29.438650 24872 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.131:44039
I20260925 04:59:29.438745 24999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.131:44039 every 8 connection(s)
I20260925 04:59:29.440032 24872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
I20260925 04:59:29.446797 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 24872
I20260925 04:59:29.447001 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance
I20260925 04:59:29.447564 6262 external_mini_cluster.cc:1428] Setting key 43000ccc6cd3d290b2407bb2572197f6
Sep 25 04:59:29 dist-test-slave-xzjr krb5kdc[24498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312369, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:59:29.462070 25002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:29.442646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:33573 (local address 127.6.29.190:33089)
0925 04:59:29.443183 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:29.443189 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:29.444091 (+ 902us) server_negotiation.cc:407] Connection header received
0925 04:59:29.445918 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:29.445923 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:29.445999 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:29.446147 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:29.448293 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:29.449498 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:29.451300 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:29.451690 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:29.451877 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:29.455395 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:29.455401 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:29.455404 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:59:29.455440 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:29.458378 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:29.459267 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:29.459271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:29.459274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:29.459436 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:29.459935 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:29.459939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:29.459941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:29.460275 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:29.460403 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:29.461418 (+ 1015us) server_negotiation.cc:299] Negotiation successful
0925 04:59:29.461658 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":235,"threads_started":1}
I20260925 04:59:29.463295 25003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:29.443160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:33089 (local address 127.6.29.131:33573)
0925 04:59:29.443938 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:29.443978 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:29.445623 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:29.446399 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:29.446410 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:29.447216 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:59:29.448089 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:29.448112 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:29.449631 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:29.449637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:29.451165 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:29.451184 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:29.451944 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:29.451949 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:29.452151 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:29.452894 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:59:29.452919 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:59:29.455186 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:59:29.458547 (+ 3361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:59:29.458554 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:59:29.458578 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:59:29.459061 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:59:29.459553 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:59:29.459556 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:59:29.459559 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:59:29.459756 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:59:29.460413 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:59:29.460419 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:59:29.460816 (+ 397us) client_negotiation.cc:769] Sending connection context
0925 04:59:29.461197 (+ 381us) client_negotiation.cc:240] Negotiation successful
0925 04:59:29.461775 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":271,"threads_started":1}
I20260925 04:59:29.464736 25000 heartbeater.cc:347] Connected to a master server at 127.6.29.190:33089
I20260925 04:59:29.465066 25000 heartbeater.cc:464] Registering TS with master...
I20260925 04:59:29.465814 25000 heartbeater.cc:511] Master 127.6.29.190:33089 requested a full tablet report, sending...
I20260925 04:59:29.467284 24543 ts_manager.cc:194] Registered new tserver with Master: d188c59aabdb4b218a131656b843cf79 (127.6.29.131:44039)
I20260925 04:59:29.468204 24543 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.131@KRBTEST.COM'} at 127.6.29.131:33573
I20260925 04:59:29.475976 6262 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:59:29.494036 25002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:29.480226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45700 (local address 127.6.29.190:33089)
0925 04:59:29.480421 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:29.480426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:29.480944 (+ 518us) server_negotiation.cc:407] Connection header received
0925 04:59:29.481083 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:29.481087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:29.481163 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:29.481252 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:29.482628 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:29.483572 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:29.484960 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:29.485438 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:29.485563 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:29.488461 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:29.488466 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:29.488469 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:59:29.488506 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:29.491038 (+ 2532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:29.492012 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:29.492016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:29.492019 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:29.492140 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:29.492718 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:29.492721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:29.492723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:29.493095 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:29.493332 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:29.493658 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 04:59:29.493814 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 04:59:29.498330 24543 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:59:29.505456 24543 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:59:29.546824 25013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:29.518313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:34005 (local address 127.0.0.1:38132)
0925 04:59:29.519149 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:29.519205 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:29.519373 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:29.533636 (+ 14263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:29.533649 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:29.533720 (+ 71us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:59:29.534540 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:29.534561 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:29.538300 (+ 3739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:29.538310 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:29.542786 (+ 4476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:29.542816 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:29.545607 (+ 2791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:29.545613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:29.545941 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:29.546012 (+ 71us) client_negotiation.cc:769] Sending connection context
0925 04:59:29.546151 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 04:59:29.546341 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":99,"thread_start_us":433,"threads_started":1}
I20260925 04:59:29.548242 25014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:29.518615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38132 (local address 127.6.29.129:34005)
0925 04:59:29.532392 (+ 13777us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:29.532405 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:29.532443 (+ 38us) server_negotiation.cc:407] Connection header received
0925 04:59:29.532573 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:29.532586 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:29.533220 (+ 634us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:29.533506 (+ 286us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:59:29.535856 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:29.538069 (+ 2213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:29.543081 (+ 5012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:29.545334 (+ 2253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:29.545798 (+ 464us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:29.546055 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:29.547627 (+ 1572us) server_negotiation.cc:299] Negotiation successful
0925 04:59:29.547916 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13519,"thread_start_us":14712,"threads_started":1}
I20260925 04:59:29.550832 25015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:29.519580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.131:44039 (local address 127.0.0.1:51662)
0925 04:59:29.536470 (+ 16890us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:29.536504 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:29.536658 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:29.539788 (+ 3130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:29.539797 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:29.539836 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:59:29.540498 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:29.540516 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:29.543264 (+ 2748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:29.543273 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:29.546711 (+ 3438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:29.546725 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:29.548742 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:29.548895 (+ 153us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:29.549182 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:29.549248 (+ 66us) client_negotiation.cc:769] Sending connection context
0925 04:59:29.549353 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 04:59:29.550405 (+ 1052us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16696,"mutex_wait_us":1,"thread_start_us":17178,"threads_started":1}
I20260925 04:59:29.550976 25016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:29.520113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51662 (local address 127.6.29.131:44039)
0925 04:59:29.538199 (+ 18086us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:29.538214 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:29.538289 (+ 75us) server_negotiation.cc:407] Connection header received
0925 04:59:29.538478 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:29.538491 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:29.539027 (+ 536us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:29.539307 (+ 280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:59:29.540811 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:29.543087 (+ 2276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:29.546918 (+ 3831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:29.548490 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:29.548624 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:29.548888 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:29.549473 (+ 585us) server_negotiation.cc:299] Negotiation successful
0925 04:59:29.549647 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17724,"thread_start_us":17098,"threads_started":1}
I20260925 04:59:29.553257 24667 tablet_service.cc:1467] Processing CreateTablet for tablet 79d36d7d6bd24d248d1cdbabc88d6f3e (DEFAULT_TABLE table=test-table [id=28e95c7a3a03494d9fdbe7ee9d6f7fff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:59:29.554992 24667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79d36d7d6bd24d248d1cdbabc88d6f3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:59:29.555045 24935 tablet_service.cc:1467] Processing CreateTablet for tablet 79d36d7d6bd24d248d1cdbabc88d6f3e (DEFAULT_TABLE table=test-table [id=28e95c7a3a03494d9fdbe7ee9d6f7fff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:59:29.557073 24935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79d36d7d6bd24d248d1cdbabc88d6f3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:59:29.559343 25017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:29.520290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:38157 (local address 127.0.0.1:45696)
0925 04:59:29.545494 (+ 25204us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:29.545522 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:29.545656 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:29.547292 (+ 1636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:29.547297 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:29.547330 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:59:29.547727 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:29.547743 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:29.551669 (+ 3926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:29.551674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:29.554537 (+ 2863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:29.554553 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:29.558531 (+ 3978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:29.558538 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:29.558726 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:29.558764 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 04:59:29.558849 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 04:59:29.558996 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25045,"thread_start_us":24894,"threads_started":1}
I20260925 04:59:29.560180 25018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:29.520572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45696 (local address 127.6.29.130:38157)
0925 04:59:29.544989 (+ 24417us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:29.545000 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:29.546014 (+ 1014us) server_negotiation.cc:407] Connection header received
0925 04:59:29.546146 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:29.546158 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:29.546456 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:29.546677 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:59:29.548016 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:29.551492 (+ 3476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:29.556546 (+ 5054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:29.558020 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:29.558194 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:29.558394 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:29.559650 (+ 1256us) server_negotiation.cc:299] Negotiation successful
0925 04:59:29.559855 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24236,"thread_start_us":25171,"threads_started":1}
I20260925 04:59:29.565495 24801 tablet_service.cc:1467] Processing CreateTablet for tablet 79d36d7d6bd24d248d1cdbabc88d6f3e (DEFAULT_TABLE table=test-table [id=28e95c7a3a03494d9fdbe7ee9d6f7fff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:59:29.567009 24801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79d36d7d6bd24d248d1cdbabc88d6f3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:59:29.571205 25019 tablet_bootstrap.cc:492] T 79d36d7d6bd24d248d1cdbabc88d6f3e P d188c59aabdb4b218a131656b843cf79: Bootstrap starting.
I20260925 04:59:29.575042 25020 tablet_bootstrap.cc:492] T 79d36d7d6bd24d248d1cdbabc88d6f3e P 9c63675290894ba187f03360b57ad509: Bootstrap starting.
I20260925 04:59:29.576156 25019 tablet_bootstrap.cc:654] T 79d36d7d6bd24d248d1cdbabc88d6f3e P d188c59aabdb4b218a131656b843cf79: Neither blocks nor log segments found. Creating new log.
I20260925 04:59:29.577587 25019 log.cc:824] T 79d36d7d6bd24d248d1cdbabc88d6f3e P d188c59aabdb4b218a131656b843cf79: Log is configured to *not* fsync() on all Append() calls
I20260925 04:59:29.581337 25020 tablet_bootstrap.cc:654] T 79d36d7d6bd24d248d1cdbabc88d6f3e P 9c63675290894ba187f03360b57ad509: Neither blocks nor log segments found. Creating new log.
I20260925 04:59:29.583400 25019 tablet_bootstrap.cc:492] T 79d36d7d6bd24d248d1cdbabc88d6f3e P d188c59aabdb4b218a131656b843cf79: No bootstrap required, opened a new log
I20260925 04:59:29.583789 25019 ts_tablet_manager.cc:1397] T 79d36d7d6bd24d248d1cdbabc88d6f3e P d188c59aabdb4b218a131656b843cf79: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260925 04:59:29.586853 25022 tablet_bootstrap.cc:492] T 79d36d7d6bd24d248d1cdbabc88d6f3e P a5dca8adef804dc3a5961f72846ac507: Bootstrap starting.
I20260925 04:59:29.588407 25020 log.cc:824] T 79d36d7d6bd24d248d1cdbabc88d6f3e P 9c63675290894ba187f03360b57ad509: Log is configured to *not* fsync() on all Append() calls
I20260925 04:59:29.591872 25020 tablet_bootstrap.cc:492] T 79d36d7d6bd24d248d1cdbabc88d6f3e P 9c63675290894ba187f03360b57ad509: No bootstrap required, opened a new log
I20260925 04:59:29.592146 25022 tablet_bootstrap.cc:654] T 79d36d7d6bd24d248d1cdbabc88d6f3e P a5dca8adef804dc3a5961f72846ac507: Neither blocks nor log segments found. Creating new log.
I20260925 04:59:29.592228 25020 ts_tablet_manager.cc:1397] T 79d36d7d6bd24d248d1cdbabc88d6f3e P 9c63675290894ba187f03360b57ad509: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.000s
I20260925 04:59:29.593580 25022 log.cc:824] T 79d36d7d6bd24d248d1cdbabc88d6f3e P a5dca8adef804dc3a5961f72846ac507: Log is configured to *not* fsync() on all Append() calls
I20260925 04:59:29.594375 25019 raft_consensus.cc:348] T 79d36d7d6bd24d248d1cdbabc88d6f3e P d188c59aabdb4b218a131656b843cf79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c63675290894ba187f03360b57ad509" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 34005 } } peers { permanent_uuid: "d188c59aabdb4b218a131656b843cf79" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 44039 } } peers { permanent_uuid: "a5dca8adef804dc3a5961f72846ac507" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 38157 } }
I20260925 04:59:29.594848 25019 raft_consensus.cc:374] T 79d36d7d6bd24d248d1cdbabc88d6f3e P d188c59aabdb4b218a131656b843cf79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:59:29.594950 25019 raft_consensus.cc:729] T 79d36d7d6bd24d248d1cdbabc88d6f3e P d188c59aabdb4b218a131656b843cf79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d188c59aabdb4b218a131656b843cf79, State: Initialized, Role: FOLLOWER
I20260925 04:59:29.595682 25019 consensus_queue.cc:260] T 79d36d7d6bd24d248d1cdbabc88d6f3e P d188c59aabdb4b218a131656b843cf79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c63675290894ba187f03360b57ad509" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 34005 } } peers { permanent_uuid: "d188c59aabdb4b218a131656b843cf79" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 44039 } } peers { permanent_uuid: "a5dca8adef804dc3a5961f72846ac507" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 38157 } }
I20260925 04:59:29.596889 25000 heartbeater.cc:503] Master 127.6.29.190:33089 was elected leader, sending a full tablet report...
I20260925 04:59:29.597642 25022 tablet_bootstrap.cc:492] T 79d36d7d6bd24d248d1cdbabc88d6f3e P a5dca8adef804dc3a5961f72846ac507: No bootstrap required, opened a new log
I20260925 04:59:29.597890 25019 ts_tablet_manager.cc:1428] T 79d36d7d6bd24d248d1cdbabc88d6f3e P d188c59aabdb4b218a131656b843cf79: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260925 04:59:29.598203 25022 ts_tablet_manager.cc:1397] T 79d36d7d6bd24d248d1cdbabc88d6f3e P a5dca8adef804dc3a5961f72846ac507: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s
I20260925 04:59:29.603919 25020 raft_consensus.cc:348] T 79d36d7d6bd24d248d1cdbabc88d6f3e P 9c63675290894ba187f03360b57ad509 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c63675290894ba187f03360b57ad509" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 34005 } } peers { permanent_uuid: "d188c59aabdb4b218a131656b843cf79" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 44039 } } peers { permanent_uuid: "a5dca8adef804dc3a5961f72846ac507" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 38157 } }
I20260925 04:59:29.604305 25020 raft_consensus.cc:374] T 79d36d7d6bd24d248d1cdbabc88d6f3e P 9c63675290894ba187f03360b57ad509 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:59:29.604413 25020 raft_consensus.cc:729] T 79d36d7d6bd24d248d1cdbabc88d6f3e P 9c63675290894ba187f03360b57ad509 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c63675290894ba187f03360b57ad509, State: Initialized, Role: FOLLOWER
I20260925 04:59:29.605062 25020 consensus_queue.cc:260] T 79d36d7d6bd24d248d1cdbabc88d6f3e P 9c63675290894ba187f03360b57ad509 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c63675290894ba187f03360b57ad509" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 34005 } } peers { permanent_uuid: "d188c59aabdb4b218a131656b843cf79" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 44039 } } peers { permanent_uuid: "a5dca8adef804dc3a5961f72846ac507" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 38157 } }
I20260925 04:59:29.606848 25020 ts_tablet_manager.cc:1428] T 79d36d7d6bd24d248d1cdbabc88d6f3e P 9c63675290894ba187f03360b57ad509: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260925 04:59:29.608590 25022 raft_consensus.cc:348] T 79d36d7d6bd24d248d1cdbabc88d6f3e P a5dca8adef804dc3a5961f72846ac507 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c63675290894ba187f03360b57ad509" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 34005 } } peers { permanent_uuid: "d188c59aabdb4b218a131656b843cf79" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 44039 } } peers { permanent_uuid: "a5dca8adef804dc3a5961f72846ac507" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 38157 } }
I20260925 04:59:29.609035 25022 raft_consensus.cc:374] T 79d36d7d6bd24d248d1cdbabc88d6f3e P a5dca8adef804dc3a5961f72846ac507 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:59:29.609165 25022 raft_consensus.cc:729] T 79d36d7d6bd24d248d1cdbabc88d6f3e P a5dca8adef804dc3a5961f72846ac507 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5dca8adef804dc3a5961f72846ac507, State: Initialized, Role: FOLLOWER
I20260925 04:59:29.609737 25022 consensus_queue.cc:260] T 79d36d7d6bd24d248d1cdbabc88d6f3e P a5dca8adef804dc3a5961f72846ac507 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c63675290894ba187f03360b57ad509" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 34005 } } peers { permanent_uuid: "d188c59aabdb4b218a131656b843cf79" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 44039 } } peers { permanent_uuid: "a5dca8adef804dc3a5961f72846ac507" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 38157 } }
I20260925 04:59:29.611001 24866 heartbeater.cc:503] Master 127.6.29.190:33089 was elected leader, sending a full tablet report...
I20260925 04:59:29.611640 25022 ts_tablet_manager.cc:1428] T 79d36d7d6bd24d248d1cdbabc88d6f3e P a5dca8adef804dc3a5961f72846ac507: Time spent starting tablet: real 0.013s user 0.006s sys 0.005s
W20260925 04:59:29.615952 24733 tablet.cc:2369] T 79d36d7d6bd24d248d1cdbabc88d6f3e P 9c63675290894ba187f03360b57ad509: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:59:29.637035 24867 tablet.cc:2369] T 79d36d7d6bd24d248d1cdbabc88d6f3e P a5dca8adef804dc3a5961f72846ac507: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:59:29.691725 25001 tablet.cc:2369] T 79d36d7d6bd24d248d1cdbabc88d6f3e P d188c59aabdb4b218a131656b843cf79: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:59:29.922315 25026 raft_consensus.cc:482] T 79d36d7d6bd24d248d1cdbabc88d6f3e P 9c63675290894ba187f03360b57ad509 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:59:29.922607 25026 raft_consensus.cc:504] T 79d36d7d6bd24d248d1cdbabc88d6f3e P 9c63675290894ba187f03360b57ad509 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c63675290894ba187f03360b57ad509" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 34005 } } peers { permanent_uuid: "d188c59aabdb4b218a131656b843cf79" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 44039 } } peers { permanent_uuid: "a5dca8adef804dc3a5961f72846ac507" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 38157 } }
I20260925 04:59:29.924474 25026 leader_election.cc:288] T 79d36d7d6bd24d248d1cdbabc88d6f3e P 9c63675290894ba187f03360b57ad509 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d188c59aabdb4b218a131656b843cf79 (127.6.29.131:44039), a5dca8adef804dc3a5961f72846ac507 (127.6.29.130:38157)
I20260925 04:59:29.931943 25018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:29.925577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:51985 (local address 127.6.29.130:38157)
0925 04:59:29.925813 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:29.925821 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:29.925958 (+ 137us) server_negotiation.cc:407] Connection header received
0925 04:59:29.926221 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:29.926226 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:29.926306 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:29.926396 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:59:29.926994 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:29.927824 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:29.929900 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:29.930883 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:29.931003 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:29.931059 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:29.931485 (+ 426us) server_negotiation.cc:299] Negotiation successful
0925 04:59:29.931651 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 04:59:29.932190 25016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:29.924871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:47525 (local address 127.6.29.131:44039)
0925 04:59:29.925086 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:29.925094 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:29.926099 (+ 1005us) server_negotiation.cc:407] Connection header received
0925 04:59:29.926327 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:29.926335 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:29.926411 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:29.926489 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:59:29.927270 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:29.928327 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:29.930285 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:29.931174 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:29.931319 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:29.931397 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:29.931817 (+ 420us) server_negotiation.cc:299] Negotiation successful
0925 04:59:29.931955 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 04:59:29.932272 25028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:29.924994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:38157 (local address 127.6.29.129:51985)
0925 04:59:29.925839 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:29.925863 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:29.926044 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:29.926415 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:29.926452 (+ 37us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:29.926480 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:59:29.926853 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:29.926863 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:29.928065 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:29.928071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:29.929734 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:29.929745 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:29.931087 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:29.931095 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:29.931281 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:29.931332 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 04:59:29.931442 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 04:59:29.931556 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"mutex_wait_us":46,"thread_start_us":829,"threads_started":1}
I20260925 04:59:29.932614 25029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:29.925023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.131:44039 (local address 127.6.29.129:47525)
0925 04:59:29.925986 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:29.926005 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:29.926202 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:29.926643 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:29.926646 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:29.926666 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:59:29.927051 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:29.927065 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:29.928496 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:29.928500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:29.930128 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:29.930142 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:29.931385 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:29.931430 (+ 45us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:29.931600 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:29.931632 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 04:59:29.931725 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 04:59:29.931890 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":838,"mutex_wait_us":1,"thread_start_us":503,"threads_started":1}
I20260925 04:59:29.933393 24821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79d36d7d6bd24d248d1cdbabc88d6f3e" candidate_uuid: "9c63675290894ba187f03360b57ad509" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5dca8adef804dc3a5961f72846ac507" is_pre_election: true
I20260925 04:59:29.933766 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79d36d7d6bd24d248d1cdbabc88d6f3e" candidate_uuid: "9c63675290894ba187f03360b57ad509" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d188c59aabdb4b218a131656b843cf79" is_pre_election: true
I20260925 04:59:29.934015 24821 raft_consensus.cc:2454] T 79d36d7d6bd24d248d1cdbabc88d6f3e P a5dca8adef804dc3a5961f72846ac507 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c63675290894ba187f03360b57ad509 in term 0.
I20260925 04:59:29.934266 24955 raft_consensus.cc:2454] T 79d36d7d6bd24d248d1cdbabc88d6f3e P d188c59aabdb4b218a131656b843cf79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c63675290894ba187f03360b57ad509 in term 0.
I20260925 04:59:29.934830 24621 leader_election.cc:302] T 79d36d7d6bd24d248d1cdbabc88d6f3e P 9c63675290894ba187f03360b57ad509 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 9c63675290894ba187f03360b57ad509, a5dca8adef804dc3a5961f72846ac507, d188c59aabdb4b218a131656b843cf79; no voters:
I20260925 04:59:29.935248 25026 raft_consensus.cc:2781] T 79d36d7d6bd24d248d1cdbabc88d6f3e P 9c63675290894ba187f03360b57ad509 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:59:29.935396 25026 raft_consensus.cc:482] T 79d36d7d6bd24d248d1cdbabc88d6f3e P 9c63675290894ba187f03360b57ad509 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:59:29.935490 25026 raft_consensus.cc:3037] T 79d36d7d6bd24d248d1cdbabc88d6f3e P 9c63675290894ba187f03360b57ad509 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:59:29.937763 25026 raft_consensus.cc:504] T 79d36d7d6bd24d248d1cdbabc88d6f3e P 9c63675290894ba187f03360b57ad509 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c63675290894ba187f03360b57ad509" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 34005 } } peers { permanent_uuid: "d188c59aabdb4b218a131656b843cf79" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 44039 } } peers { permanent_uuid: "a5dca8adef804dc3a5961f72846ac507" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 38157 } }
I20260925 04:59:29.938347 25026 leader_election.cc:288] T 79d36d7d6bd24d248d1cdbabc88d6f3e P 9c63675290894ba187f03360b57ad509 [CANDIDATE]: Term 1 election: Requested vote from peers d188c59aabdb4b218a131656b843cf79 (127.6.29.131:44039), a5dca8adef804dc3a5961f72846ac507 (127.6.29.130:38157)
I20260925 04:59:29.938853 24821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79d36d7d6bd24d248d1cdbabc88d6f3e" candidate_uuid: "9c63675290894ba187f03360b57ad509" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5dca8adef804dc3a5961f72846ac507"
I20260925 04:59:29.938853 24955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79d36d7d6bd24d248d1cdbabc88d6f3e" candidate_uuid: "9c63675290894ba187f03360b57ad509" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d188c59aabdb4b218a131656b843cf79"
I20260925 04:59:29.939172 24821 raft_consensus.cc:3037] T 79d36d7d6bd24d248d1cdbabc88d6f3e P a5dca8adef804dc3a5961f72846ac507 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:59:29.939150 24955 raft_consensus.cc:3037] T 79d36d7d6bd24d248d1cdbabc88d6f3e P d188c59aabdb4b218a131656b843cf79 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:59:29.942481 24821 raft_consensus.cc:2454] T 79d36d7d6bd24d248d1cdbabc88d6f3e P a5dca8adef804dc3a5961f72846ac507 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c63675290894ba187f03360b57ad509 in term 1.
I20260925 04:59:29.942479 24955 raft_consensus.cc:2454] T 79d36d7d6bd24d248d1cdbabc88d6f3e P d188c59aabdb4b218a131656b843cf79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c63675290894ba187f03360b57ad509 in term 1.
I20260925 04:59:29.942919 24621 leader_election.cc:302] T 79d36d7d6bd24d248d1cdbabc88d6f3e P 9c63675290894ba187f03360b57ad509 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c63675290894ba187f03360b57ad509, a5dca8adef804dc3a5961f72846ac507; no voters:
I20260925 04:59:29.943228 25026 raft_consensus.cc:2781] T 79d36d7d6bd24d248d1cdbabc88d6f3e P 9c63675290894ba187f03360b57ad509 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:59:29.943665 25026 raft_consensus.cc:686] T 79d36d7d6bd24d248d1cdbabc88d6f3e P 9c63675290894ba187f03360b57ad509 [term 1 LEADER]: Becoming Leader. State: Replica: 9c63675290894ba187f03360b57ad509, State: Running, Role: LEADER
I20260925 04:59:29.944211 25026 consensus_queue.cc:237] T 79d36d7d6bd24d248d1cdbabc88d6f3e P 9c63675290894ba187f03360b57ad509 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c63675290894ba187f03360b57ad509" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 34005 } } peers { permanent_uuid: "d188c59aabdb4b218a131656b843cf79" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 44039 } } peers { permanent_uuid: "a5dca8adef804dc3a5961f72846ac507" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 38157 } }
I20260925 04:59:29.948791 24542 catalog_manager.cc:5495] T 79d36d7d6bd24d248d1cdbabc88d6f3e P 9c63675290894ba187f03360b57ad509 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c63675290894ba187f03360b57ad509 (127.6.29.129). New cstate: current_term: 1 leader_uuid: "9c63675290894ba187f03360b57ad509" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c63675290894ba187f03360b57ad509" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 34005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d188c59aabdb4b218a131656b843cf79" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 44039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5dca8adef804dc3a5961f72846ac507" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 38157 } health_report { overall_health: UNKNOWN } } }
I20260925 04:59:30.019404 25014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:30.013430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38146 (local address 127.6.29.129:34005)
0925 04:59:30.013627 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:30.013631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:30.014097 (+ 466us) server_negotiation.cc:407] Connection header received
0925 04:59:30.014269 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:30.014273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:30.014354 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:30.014494 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:59:30.014974 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:30.015704 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:30.017205 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:30.017614 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:30.017721 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:30.018281 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:59:30.018816 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:59:30.018963 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:30.019119 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 04:59:30.019161 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 04:59:30.028312 24955 raft_consensus.cc:1261] T 79d36d7d6bd24d248d1cdbabc88d6f3e P d188c59aabdb4b218a131656b843cf79 [term 1 FOLLOWER]: Refusing update from remote peer 9c63675290894ba187f03360b57ad509: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:59:30.028312 24821 raft_consensus.cc:1261] T 79d36d7d6bd24d248d1cdbabc88d6f3e P a5dca8adef804dc3a5961f72846ac507 [term 1 FOLLOWER]: Refusing update from remote peer 9c63675290894ba187f03360b57ad509: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:59:30.029532 25030 consensus_queue.cc:1035] T 79d36d7d6bd24d248d1cdbabc88d6f3e P 9c63675290894ba187f03360b57ad509 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5dca8adef804dc3a5961f72846ac507" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 38157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:59:30.029920 25026 consensus_queue.cc:1035] T 79d36d7d6bd24d248d1cdbabc88d6f3e P 9c63675290894ba187f03360b57ad509 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d188c59aabdb4b218a131656b843cf79" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 44039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:59:30.041501 25034 mvcc.cc:205] Tried to move back new op lower bound from 7333119467623751680 to 7333119467295498240. Current Snapshot: MvccSnapshot[applied={T|T < 7333119467623751680}]
I20260925 04:59:30.056275 24543 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:59:30.056530 24543 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:59:30.060123 24543 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f2481b34bd954464a93d56cd9cc99bbc: Sending DeleteTablet for 3 replicas of tablet 79d36d7d6bd24d248d1cdbabc88d6f3e
I20260925 04:59:30.061307 24667 tablet_service.cc:1514] Processing DeleteTablet for tablet 79d36d7d6bd24d248d1cdbabc88d6f3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:59:30 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:38132
I20260925 04:59:30.061481 24935 tablet_service.cc:1514] Processing DeleteTablet for tablet 79d36d7d6bd24d248d1cdbabc88d6f3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:59:30 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:51662
I20260925 04:59:30.061476 24801 tablet_service.cc:1514] Processing DeleteTablet for tablet 79d36d7d6bd24d248d1cdbabc88d6f3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:59:30 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:45696
I20260925 04:59:30.068540 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 24604
I20260925 04:59:30.072377 25044 tablet_replica.cc:330] T 79d36d7d6bd24d248d1cdbabc88d6f3e P d188c59aabdb4b218a131656b843cf79: stopping tablet replica
I20260925 04:59:30.072978 25044 raft_consensus.cc:2229] T 79d36d7d6bd24d248d1cdbabc88d6f3e P d188c59aabdb4b218a131656b843cf79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:59:30.073650 25044 raft_consensus.cc:2258] T 79d36d7d6bd24d248d1cdbabc88d6f3e P d188c59aabdb4b218a131656b843cf79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:59:30.079950 25044 ts_tablet_manager.cc:1905] T 79d36d7d6bd24d248d1cdbabc88d6f3e P d188c59aabdb4b218a131656b843cf79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:59:30.097582 25043 tablet_replica.cc:330] T 79d36d7d6bd24d248d1cdbabc88d6f3e P 9c63675290894ba187f03360b57ad509: stopping tablet replica
I20260925 04:59:30.097808 25045 tablet_replica.cc:330] T 79d36d7d6bd24d248d1cdbabc88d6f3e P a5dca8adef804dc3a5961f72846ac507: stopping tablet replica
I20260925 04:59:30.098345 25043 raft_consensus.cc:2229] T 79d36d7d6bd24d248d1cdbabc88d6f3e P 9c63675290894ba187f03360b57ad509 [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:59:30.098345 25045 raft_consensus.cc:2229] T 79d36d7d6bd24d248d1cdbabc88d6f3e P a5dca8adef804dc3a5961f72846ac507 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:59:30.099038 25045 raft_consensus.cc:2258] T 79d36d7d6bd24d248d1cdbabc88d6f3e P a5dca8adef804dc3a5961f72846ac507 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:59:30.099752 25043 raft_consensus.cc:2258] T 79d36d7d6bd24d248d1cdbabc88d6f3e P 9c63675290894ba187f03360b57ad509 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:59:30.100087 25044 ts_tablet_manager.cc:1918] T 79d36d7d6bd24d248d1cdbabc88d6f3e P d188c59aabdb4b218a131656b843cf79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:59:30.100287 25044 log.cc:1196] T 79d36d7d6bd24d248d1cdbabc88d6f3e P d188c59aabdb4b218a131656b843cf79: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-2/wal/wals/79d36d7d6bd24d248d1cdbabc88d6f3e
I20260925 04:59:30.100841 25044 ts_tablet_manager.cc:1939] T 79d36d7d6bd24d248d1cdbabc88d6f3e P d188c59aabdb4b218a131656b843cf79: Deleting consensus metadata
I20260925 04:59:30.101301 25045 ts_tablet_manager.cc:1905] T 79d36d7d6bd24d248d1cdbabc88d6f3e P a5dca8adef804dc3a5961f72846ac507: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 04:59:30 dist-test-slave-xzjr krb5kdc[24498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.129@KRBTEST.COM
I20260925 04:59:30.102946 24527 catalog_manager.cc:4841] TS d188c59aabdb4b218a131656b843cf79 (127.6.29.131:44039): tablet 79d36d7d6bd24d248d1cdbabc88d6f3e (table test-table [id=28e95c7a3a03494d9fdbe7ee9d6f7fff]) successfully deleted
I20260925 04:59:30.104301 25043 ts_tablet_manager.cc:1905] T 79d36d7d6bd24d248d1cdbabc88d6f3e P 9c63675290894ba187f03360b57ad509: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:59:30.108464 25045 ts_tablet_manager.cc:1918] T 79d36d7d6bd24d248d1cdbabc88d6f3e P a5dca8adef804dc3a5961f72846ac507: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:59:30.108662 25045 log.cc:1196] T 79d36d7d6bd24d248d1cdbabc88d6f3e P a5dca8adef804dc3a5961f72846ac507: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-1/wal/wals/79d36d7d6bd24d248d1cdbabc88d6f3e
I20260925 04:59:30.109207 25045 ts_tablet_manager.cc:1939] T 79d36d7d6bd24d248d1cdbabc88d6f3e P a5dca8adef804dc3a5961f72846ac507: Deleting consensus metadata
I20260925 04:59:30.111095 24528 catalog_manager.cc:4841] TS a5dca8adef804dc3a5961f72846ac507 (127.6.29.130:38157): tablet 79d36d7d6bd24d248d1cdbabc88d6f3e (table test-table [id=28e95c7a3a03494d9fdbe7ee9d6f7fff]) successfully deleted
I20260925 04:59:30.112293 25014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:30.070924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38160 (local address 127.6.29.129:34005)
0925 04:59:30.071207 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:30.071216 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:30.074536 (+ 3320us) server_negotiation.cc:407] Connection header received
0925 04:59:30.074763 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:30.074772 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:30.074888 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:30.075021 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:30.080307 (+ 5286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:30.081963 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:30.092320 (+ 10357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:30.093424 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:30.093572 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:30.103876 (+ 10304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:30.103891 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:30.103909 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:59:30.103956 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:30.108429 (+ 4473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:30.109385 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:30.109394 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:30.109398 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:30.109580 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:30.110094 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:30.110100 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:30.110103 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:30.110618 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:30.110965 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:30.111606 (+ 641us) server_negotiation.cc:299] Negotiation successful
0925 04:59:30.111916 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 04:59:30.113054 24727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:59:30.213193 25043 ts_tablet_manager.cc:1918] T 79d36d7d6bd24d248d1cdbabc88d6f3e P 9c63675290894ba187f03360b57ad509: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:59:30.213476 25043 log.cc:1196] T 79d36d7d6bd24d248d1cdbabc88d6f3e P 9c63675290894ba187f03360b57ad509: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790312053666503-6262-0/minicluster-data/ts-0/wal/wals/79d36d7d6bd24d248d1cdbabc88d6f3e
I20260925 04:59:30.213824 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 24604
I20260925 04:59:30.236725 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 24738
W20260925 04:59:30.236734 24527 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.29.129:34005 (error 108)
I20260925 04:59:30.238732 25049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:30.237965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:34005 (local address 127.0.0.1:38174)
0925 04:59:30.238438 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:30.238547 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.129:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":208,"threads_started":1}
W20260925 04:59:30.238963 24527 catalog_manager.cc:4584] TS 9c63675290894ba187f03360b57ad509 (127.6.29.129:34005): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 79d36d7d6bd24d248d1cdbabc88d6f3e: Network error: Client connection negotiation failed: client connection to 127.6.29.129:34005: connect: Connection refused (error 111)
Sep 25 04:59:30 dist-test-slave-xzjr krb5kdc[24498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.130@KRBTEST.COM
I20260925 04:59:30.252133 25018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:30.237522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45700 (local address 127.6.29.130:38157)
0925 04:59:30.237705 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:30.237711 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:30.237796 (+ 85us) server_negotiation.cc:407] Connection header received
0925 04:59:30.238003 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:30.238009 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:30.238142 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:30.238259 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:30.239676 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:30.240483 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:30.241800 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:30.242266 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:30.242369 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:30.246641 (+ 4272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:30.246656 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:30.246672 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:59:30.246714 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:30.249646 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:30.250288 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:30.250295 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:30.250297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:30.250421 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:30.250743 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:30.250746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:30.250748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:30.251117 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:30.251330 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:30.251553 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 04:59:30.251708 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 04:59:30.252781 24861 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:59:30.292572 25049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:30.292122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:34005 (local address 127.0.0.1:38182)
0925 04:59:30.292328 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:30.292430 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.129:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 04:59:30.293663 25049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:30.293286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:34005 (local address 127.0.0.1:38188)
0925 04:59:30.293454 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:30.293528 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.129:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 04:59:30.342622 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 24738
I20260925 04:59:30.354240 25049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:30.353782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:34005 (local address 127.0.0.1:38190)
0925 04:59:30.353996 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:30.354087 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.129:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 04:59:30.355218 25049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:30.354881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:34005 (local address 127.0.0.1:38198)
0925 04:59:30.355019 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:30.355088 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.129:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 04:59:30.360097 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 24872
Sep 25 04:59:30 dist-test-slave-xzjr krb5kdc[24498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.131@KRBTEST.COM
I20260925 04:59:30.375716 25016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:30.360862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51668 (local address 127.6.29.131:44039)
0925 04:59:30.361181 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:30.361188 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:30.361289 (+ 101us) server_negotiation.cc:407] Connection header received
0925 04:59:30.361502 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:30.361507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:30.361621 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:30.361729 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:30.362993 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:30.363731 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:30.365029 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:30.365428 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:30.365562 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:30.370046 (+ 4484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:30.370061 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:30.370077 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:59:30.370112 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:30.372897 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:30.373610 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:30.373618 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:30.373620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:30.373757 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:30.374130 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:30.374139 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:30.374142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:30.374502 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:30.374826 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:30.375109 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 04:59:30.375336 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186}
I20260925 04:59:30.376497 24995 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:59:30.436462 25049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:30.435965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:34005 (local address 127.0.0.1:38210)
0925 04:59:30.436203 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:30.436301 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.129:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260925 04:59:30.437562 25049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:30.437161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:34005 (local address 127.0.0.1:38216)
0925 04:59:30.437372 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:30.437438 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.129:34005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260925 04:59:30.467244 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 24872
I20260925 04:59:30.484040 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 24511
I20260925 04:59:30.485275 24573 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:59:30.599435 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 24511
2026-09-25T04:59:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4687 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T04:59:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:59:30Z Disabled control of system clock
I20260925 04:59:30.677701 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:45411
--webserver_interface=127.6.29.190
--webserver_port=0
--builtin_ntp_servers=127.6.29.148:36295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:45411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 04:59:31.020658 25056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:59:31.020985 25056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:59:31.021134 25056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:59:31.028733 25056 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:59:31.028864 25056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:59:31.028944 25056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:59:31.029055 25056 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:59:31.029150 25056 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:59:31.038916 25056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:36295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:45411
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:45411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:59:31.040403 25056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:59:31.042423 25056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:59:31.054795 25061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:31.054862 25062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:31.054884 25064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:59:31.056046 25056 server_base.cc:1034] running on GCE node
I20260925 04:59:31.056900 25056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:59:31.058499 25056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:59:31.059746 25056 hybrid_clock.cc:648] HybridClock initialized: now 1790312371059704 us; error 31 us; skew 500 ppm
I20260925 04:59:31.060156 25056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 04:59:31.062492 25056 webserver.cc:460] Webserver started at http://127.6.29.190:44129/ using document root <none> and password file <none>
I20260925 04:59:31.063158 25056 fs_manager.cc:359] Metadata directory not provided
I20260925 04:59:31.063256 25056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:59:31.063544 25056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:59:31.066579 25056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "bae37a2535d640eea6da537f180e51ad"
format_stamp: "Formatted at 2026-09-25 04:59:31 on dist-test-slave-xzjr"
server_key: "71b7ab8d54acf5084349c4e42429e301"
server_key_iv: "aa872b828c7c48c766df6a905b4bdaa2"
server_key_version: "encryptionkey@0"
I20260925 04:59:31.067363 25056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "bae37a2535d640eea6da537f180e51ad"
format_stamp: "Formatted at 2026-09-25 04:59:31 on dist-test-slave-xzjr"
server_key: "71b7ab8d54acf5084349c4e42429e301"
server_key_iv: "aa872b828c7c48c766df6a905b4bdaa2"
server_key_version: "encryptionkey@0"
I20260925 04:59:31.072896 25056 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 04:59:31.076539 25070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:31.078011 25056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:59:31.078265 25056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "bae37a2535d640eea6da537f180e51ad"
format_stamp: "Formatted at 2026-09-25 04:59:31 on dist-test-slave-xzjr"
server_key: "71b7ab8d54acf5084349c4e42429e301"
server_key_iv: "aa872b828c7c48c766df6a905b4bdaa2"
server_key_version: "encryptionkey@0"
I20260925 04:59:31.078495 25056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:59:31.108708 25056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:59:31.109628 25056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:59:31.109932 25056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:59:31.129125 25121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:45411 every 8 connection(s)
I20260925 04:59:31.129096 25056 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:45411
I20260925 04:59:31.130522 25056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:59:31.132787 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 25056
I20260925 04:59:31.132973 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 04:59:31.133471 6262 external_mini_cluster.cc:1428] Setting key 5b9d81a77e86df226963eece0e03c92b
I20260925 04:59:31.134840 25122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:59:31.147305 25125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:31.135340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42572 (local address 127.6.29.190:45411)
0925 04:59:31.136309 (+ 969us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:31.136320 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:31.136360 (+ 40us) server_negotiation.cc:407] Connection header received
0925 04:59:31.137934 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:31.137955 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:31.138448 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:31.138777 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:31.139444 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:31.140837 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:31.142763 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:31.143444 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:31.143571 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:31.144911 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:31.144921 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:31.144936 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:59:31.145533 (+ 597us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:31.145619 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:59:31.145643 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:31.145731 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:31.145980 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 04:59:31.146356 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":372,"threads_started":1}
I20260925 04:59:31.148765 25122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bae37a2535d640eea6da537f180e51ad: Bootstrap starting.
I20260925 04:59:31.152055 25122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bae37a2535d640eea6da537f180e51ad: Neither blocks nor log segments found. Creating new log.
I20260925 04:59:31.153163 25122 log.cc:824] T 00000000000000000000000000000000 P bae37a2535d640eea6da537f180e51ad: Log is configured to *not* fsync() on all Append() calls
I20260925 04:59:31.155886 25122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bae37a2535d640eea6da537f180e51ad: No bootstrap required, opened a new log
I20260925 04:59:31.163107 25122 raft_consensus.cc:348] T 00000000000000000000000000000000 P bae37a2535d640eea6da537f180e51ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae37a2535d640eea6da537f180e51ad" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 45411 } }
I20260925 04:59:31.163497 25122 raft_consensus.cc:374] T 00000000000000000000000000000000 P bae37a2535d640eea6da537f180e51ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:59:31.163604 25122 raft_consensus.cc:729] T 00000000000000000000000000000000 P bae37a2535d640eea6da537f180e51ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bae37a2535d640eea6da537f180e51ad, State: Initialized, Role: FOLLOWER
I20260925 04:59:31.164213 25122 consensus_queue.cc:260] T 00000000000000000000000000000000 P bae37a2535d640eea6da537f180e51ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae37a2535d640eea6da537f180e51ad" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 45411 } }
I20260925 04:59:31.164461 25122 raft_consensus.cc:388] T 00000000000000000000000000000000 P bae37a2535d640eea6da537f180e51ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:59:31.164598 25122 raft_consensus.cc:482] T 00000000000000000000000000000000 P bae37a2535d640eea6da537f180e51ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:59:31.164780 25122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bae37a2535d640eea6da537f180e51ad [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:59:31.166913 25122 raft_consensus.cc:504] T 00000000000000000000000000000000 P bae37a2535d640eea6da537f180e51ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae37a2535d640eea6da537f180e51ad" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 45411 } }
I20260925 04:59:31.167428 25122 leader_election.cc:302] T 00000000000000000000000000000000 P bae37a2535d640eea6da537f180e51ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bae37a2535d640eea6da537f180e51ad; no voters:
I20260925 04:59:31.168121 25122 leader_election.cc:288] T 00000000000000000000000000000000 P bae37a2535d640eea6da537f180e51ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:59:31.168270 25127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bae37a2535d640eea6da537f180e51ad [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:59:31.169544 25127 raft_consensus.cc:686] T 00000000000000000000000000000000 P bae37a2535d640eea6da537f180e51ad [term 1 LEADER]: Becoming Leader. State: Replica: bae37a2535d640eea6da537f180e51ad, State: Running, Role: LEADER
I20260925 04:59:31.170053 25127 consensus_queue.cc:237] T 00000000000000000000000000000000 P bae37a2535d640eea6da537f180e51ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae37a2535d640eea6da537f180e51ad" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 45411 } }
I20260925 04:59:31.170228 25122 sys_catalog.cc:564] T 00000000000000000000000000000000 P bae37a2535d640eea6da537f180e51ad [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:59:31.172907 25129 sys_catalog.cc:455] T 00000000000000000000000000000000 P bae37a2535d640eea6da537f180e51ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader bae37a2535d640eea6da537f180e51ad. Latest consensus state: current_term: 1 leader_uuid: "bae37a2535d640eea6da537f180e51ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae37a2535d640eea6da537f180e51ad" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 45411 } } }
I20260925 04:59:31.173305 25129 sys_catalog.cc:458] T 00000000000000000000000000000000 P bae37a2535d640eea6da537f180e51ad [sys.catalog]: This master's current role is: LEADER
I20260925 04:59:31.173010 25128 sys_catalog.cc:455] T 00000000000000000000000000000000 P bae37a2535d640eea6da537f180e51ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bae37a2535d640eea6da537f180e51ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bae37a2535d640eea6da537f180e51ad" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 45411 } } }
I20260925 04:59:31.173663 25128 sys_catalog.cc:458] T 00000000000000000000000000000000 P bae37a2535d640eea6da537f180e51ad [sys.catalog]: This master's current role is: LEADER
I20260925 04:59:31.175225 25136 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:59:31.179659 25136 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:59:31.188730 25136 catalog_manager.cc:1347] Generated new cluster ID: 64209109e4d84a3f9c4b9af7a5ffc560
I20260925 04:59:31.188848 25136 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:59:31.220441 25136 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:59:31.221764 25136 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:59:31.226845 25136 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bae37a2535d640eea6da537f180e51ad: Generated new TSK 0
I20260925 04:59:31.227828 25136 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:59:31.246086 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.129:0
--local_ip_for_outbound_sockets=127.6.29.129
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:45411
--builtin_ntp_servers=127.6.29.148:36295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 04:59:31.577431 25146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:59:31.577725 25146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:59:31.577860 25146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:59:31.585562 25146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:59:31.585675 25146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:59:31.585863 25146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.129
I20260925 04:59:31.595306 25146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:36295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:45411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.129
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:59:31.596701 25146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:59:31.598549 25146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:59:31.612322 25151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:31.612344 25154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:31.612270 25152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:59:31.613153 25146 server_base.cc:1034] running on GCE node
I20260925 04:59:31.613783 25146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:59:31.614789 25146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:59:31.616004 25146 hybrid_clock.cc:648] HybridClock initialized: now 1790312371615959 us; error 41 us; skew 500 ppm
I20260925 04:59:31.616326 25146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 04:59:31.618570 25146 webserver.cc:460] Webserver started at http://127.6.29.129:40901/ using document root <none> and password file <none>
I20260925 04:59:31.619307 25146 fs_manager.cc:359] Metadata directory not provided
I20260925 04:59:31.619436 25146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:59:31.619849 25146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:59:31.622736 25146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-0/data/instance:
uuid: "3acf6f27ec8b446c85747c62cadd6fd4"
format_stamp: "Formatted at 2026-09-25 04:59:31 on dist-test-slave-xzjr"
server_key: "536f8bedbc5b0f735dfac5261b7940c7"
server_key_iv: "01253688663f41db28cf3e7235355304"
server_key_version: "encryptionkey@0"
I20260925 04:59:31.623535 25146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance:
uuid: "3acf6f27ec8b446c85747c62cadd6fd4"
format_stamp: "Formatted at 2026-09-25 04:59:31 on dist-test-slave-xzjr"
server_key: "536f8bedbc5b0f735dfac5261b7940c7"
server_key_iv: "01253688663f41db28cf3e7235355304"
server_key_version: "encryptionkey@0"
I20260925 04:59:31.628775 25146 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260925 04:59:31.632422 25160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:31.633852 25146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 04:59:31.633985 25146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-0/wal
uuid: "3acf6f27ec8b446c85747c62cadd6fd4"
format_stamp: "Formatted at 2026-09-25 04:59:31 on dist-test-slave-xzjr"
server_key: "536f8bedbc5b0f735dfac5261b7940c7"
server_key_iv: "01253688663f41db28cf3e7235355304"
server_key_version: "encryptionkey@0"
I20260925 04:59:31.634171 25146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:59:31.646860 25146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:59:31.647656 25146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:59:31.648008 25146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:59:31.648965 25146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:59:31.650538 25146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:59:31.650650 25146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:31.650776 25146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:59:31.650869 25146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:31.684512 25146 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.129:34567
I20260925 04:59:31.684613 25272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.129:34567 every 8 connection(s)
I20260925 04:59:31.685847 25146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
I20260925 04:59:31.690968 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 25146
I20260925 04:59:31.691174 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance
I20260925 04:59:31.691625 6262 external_mini_cluster.cc:1428] Setting key 7945a1c79671255977d0ef0c31536aed
I20260925 04:59:31.694547 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.130:0
--local_ip_for_outbound_sockets=127.6.29.130
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:45411
--builtin_ntp_servers=127.6.29.148:36295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 04:59:31.699908 25275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:31.688563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:48287 (local address 127.6.29.190:45411)
0925 04:59:31.689044 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:31.689057 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:31.690384 (+ 1327us) server_negotiation.cc:407] Connection header received
0925 04:59:31.692591 (+ 2207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:31.692610 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:31.692690 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:31.692787 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:31.694248 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:31.695105 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:31.696877 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:31.697319 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:31.697461 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:31.698963 (+ 1502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:31.698969 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:31.698973 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:59:31.698992 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:31.699034 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:59:31.699049 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:31.699150 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:31.699384 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 04:59:31.699492 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":258,"threads_started":1}
I20260925 04:59:31.701018 25276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:31.689190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45411 (local address 127.6.29.129:48287)
0925 04:59:31.690220 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:31.690271 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:31.692305 (+ 2034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:31.693100 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:31.693113 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:31.693243 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:59:31.694067 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:31.694088 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:31.695265 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:31.695270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:31.696720 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:31.696733 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:31.697556 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:31.697566 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:31.697826 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:31.698614 (+ 788us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:59:31.698617 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:59:31.698661 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:59:31.698749 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:59:31.699149 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:59:31.699164 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:59:31.699176 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 04:59:31.699373 (+ 197us) client_negotiation.cc:240] Negotiation successful
0925 04:59:31.699863 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":285,"threads_started":1}
I20260925 04:59:31.702831 25273 heartbeater.cc:347] Connected to a master server at 127.6.29.190:45411
I20260925 04:59:31.703148 25273 heartbeater.cc:464] Registering TS with master...
I20260925 04:59:31.704298 25273 heartbeater.cc:511] Master 127.6.29.190:45411 requested a full tablet report, sending...
I20260925 04:59:31.706688 25087 ts_manager.cc:194] Registered new tserver with Master: 3acf6f27ec8b446c85747c62cadd6fd4 (127.6.29.129:34567)
I20260925 04:59:31.709200 25087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.29.129:48287
W20260925 04:59:32.031934 25277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:59:32.032217 25277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:59:32.032315 25277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:59:32.040026 25277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:59:32.040143 25277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:59:32.040308 25277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.130
I20260925 04:59:32.049577 25277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:36295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:45411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.130
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:59:32.050906 25277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:59:32.052765 25277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:59:32.065809 25282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:32.065846 25283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:32.065887 25285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:59:32.066862 25277 server_base.cc:1034] running on GCE node
I20260925 04:59:32.067454 25277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:59:32.068437 25277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:59:32.069681 25277 hybrid_clock.cc:648] HybridClock initialized: now 1790312372069632 us; error 54 us; skew 500 ppm
I20260925 04:59:32.070070 25277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 04:59:32.072556 25277 webserver.cc:460] Webserver started at http://127.6.29.130:39813/ using document root <none> and password file <none>
I20260925 04:59:32.073241 25277 fs_manager.cc:359] Metadata directory not provided
I20260925 04:59:32.073378 25277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:59:32.073716 25277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:59:32.076684 25277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-1/data/instance:
uuid: "69f7962ad09c412fbbfbec325c8a7bdb"
format_stamp: "Formatted at 2026-09-25 04:59:32 on dist-test-slave-xzjr"
server_key: "63c3b58f6f30662f909a2f40c1e6d287"
server_key_iv: "b3409c62bc0db8ade7cd8021eff23b6d"
server_key_version: "encryptionkey@0"
I20260925 04:59:32.077436 25277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance:
uuid: "69f7962ad09c412fbbfbec325c8a7bdb"
format_stamp: "Formatted at 2026-09-25 04:59:32 on dist-test-slave-xzjr"
server_key: "63c3b58f6f30662f909a2f40c1e6d287"
server_key_iv: "b3409c62bc0db8ade7cd8021eff23b6d"
server_key_version: "encryptionkey@0"
I20260925 04:59:32.082693 25277 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260925 04:59:32.086378 25291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:32.087751 25277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:59:32.087972 25277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-1/wal
uuid: "69f7962ad09c412fbbfbec325c8a7bdb"
format_stamp: "Formatted at 2026-09-25 04:59:32 on dist-test-slave-xzjr"
server_key: "63c3b58f6f30662f909a2f40c1e6d287"
server_key_iv: "b3409c62bc0db8ade7cd8021eff23b6d"
server_key_version: "encryptionkey@0"
I20260925 04:59:32.088207 25277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:59:32.117334 25277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:59:32.118254 25277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:59:32.118572 25277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:59:32.119567 25277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:59:32.121076 25277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:59:32.121193 25277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:32.121321 25277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:59:32.121404 25277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:32.158190 25277 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.130:46155
I20260925 04:59:32.158309 25403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.130:46155 every 8 connection(s)
I20260925 04:59:32.159477 25277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
I20260925 04:59:32.162485 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 25277
I20260925 04:59:32.162668 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance
I20260925 04:59:32.163213 6262 external_mini_cluster.cc:1428] Setting key 49e99fa5451a4c05bab0056aebccf8ad
I20260925 04:59:32.166594 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.131:0
--local_ip_for_outbound_sockets=127.6.29.131
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:45411
--builtin_ntp_servers=127.6.29.148:36295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 04:59:32.174474 25275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:32.162117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:33061 (local address 127.6.29.190:45411)
0925 04:59:32.162347 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:32.162354 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:32.164047 (+ 1693us) server_negotiation.cc:407] Connection header received
0925 04:59:32.166381 (+ 2334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:32.166386 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:32.166456 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:32.166544 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:32.168104 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:32.168833 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:32.171334 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:32.171802 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:32.171911 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:32.173583 (+ 1672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:32.173593 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:32.173600 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:59:32.173630 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:32.173664 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:59:32.173676 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:32.173786 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:32.174152 (+ 366us) server_negotiation.cc:299] Negotiation successful
0925 04:59:32.174236 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 04:59:32.175962 25406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:32.162776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45411 (local address 127.6.29.130:33061)
0925 04:59:32.163890 (+ 1114us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:32.163938 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:32.166057 (+ 2119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:32.166850 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:32.166864 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:32.166988 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:59:32.167891 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:32.167923 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:32.168979 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:32.168987 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:32.171141 (+ 2154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:32.171159 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:32.172049 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:32.172059 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:32.172414 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:32.173205 (+ 791us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:59:32.173211 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:59:32.173265 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:59:32.173368 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:59:32.173800 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:59:32.173814 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:59:32.173825 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 04:59:32.174025 (+ 200us) client_negotiation.cc:240] Negotiation successful
0925 04:59:32.174459 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":291,"threads_started":1}
I20260925 04:59:32.177359 25404 heartbeater.cc:347] Connected to a master server at 127.6.29.190:45411
I20260925 04:59:32.177680 25404 heartbeater.cc:464] Registering TS with master...
I20260925 04:59:32.178645 25404 heartbeater.cc:511] Master 127.6.29.190:45411 requested a full tablet report, sending...
I20260925 04:59:32.180227 25087 ts_manager.cc:194] Registered new tserver with Master: 69f7962ad09c412fbbfbec325c8a7bdb (127.6.29.130:46155)
I20260925 04:59:32.181022 25087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.29.130:33061
W20260925 04:59:32.500983 25407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:59:32.501299 25407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:59:32.501451 25407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:59:32.508844 25407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:59:32.508960 25407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:59:32.509141 25407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.131
I20260925 04:59:32.518400 25407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:36295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:45411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.131
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:59:32.519752 25407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:59:32.521533 25407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:59:32.535034 25412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:32.535063 25413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:32.535084 25415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:59:32.535950 25407 server_base.cc:1034] running on GCE node
I20260925 04:59:32.536602 25407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:59:32.537568 25407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:59:32.538785 25407 hybrid_clock.cc:648] HybridClock initialized: now 1790312372538735 us; error 53 us; skew 500 ppm
I20260925 04:59:32.539108 25407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 04:59:32.541469 25407 webserver.cc:460] Webserver started at http://127.6.29.131:40801/ using document root <none> and password file <none>
I20260925 04:59:32.542160 25407 fs_manager.cc:359] Metadata directory not provided
I20260925 04:59:32.542289 25407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:59:32.542642 25407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:59:32.545496 25407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-2/data/instance:
uuid: "6c106d1c02a341daa2c7f23166afd23e"
format_stamp: "Formatted at 2026-09-25 04:59:32 on dist-test-slave-xzjr"
server_key: "45a169562c5cbb8d270de5b1336f905d"
server_key_iv: "8dccf6958c8ccfc97eadbd9eb401ff56"
server_key_version: "encryptionkey@0"
I20260925 04:59:32.546281 25407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance:
uuid: "6c106d1c02a341daa2c7f23166afd23e"
format_stamp: "Formatted at 2026-09-25 04:59:32 on dist-test-slave-xzjr"
server_key: "45a169562c5cbb8d270de5b1336f905d"
server_key_iv: "8dccf6958c8ccfc97eadbd9eb401ff56"
server_key_version: "encryptionkey@0"
I20260925 04:59:32.551250 25407 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 04:59:32.554766 25421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:32.556113 25407 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 04:59:32.556277 25407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-2/wal
uuid: "6c106d1c02a341daa2c7f23166afd23e"
format_stamp: "Formatted at 2026-09-25 04:59:32 on dist-test-slave-xzjr"
server_key: "45a169562c5cbb8d270de5b1336f905d"
server_key_iv: "8dccf6958c8ccfc97eadbd9eb401ff56"
server_key_version: "encryptionkey@0"
I20260925 04:59:32.556486 25407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:59:32.584996 25407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:59:32.586033 25407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:59:32.586341 25407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:59:32.587414 25407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:59:32.588966 25407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:59:32.589093 25407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:32.589213 25407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:59:32.589296 25407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:32.622064 25407 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.131:39117
I20260925 04:59:32.622125 25533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.131:39117 every 8 connection(s)
I20260925 04:59:32.623366 25407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
I20260925 04:59:32.625245 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 25407
I20260925 04:59:32.625442 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance
I20260925 04:59:32.625924 6262 external_mini_cluster.cc:1428] Setting key 6f8b437c067691a70d27cf9b1945ba77
I20260925 04:59:32.636094 25275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:32.626191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:49533 (local address 127.6.29.190:45411)
0925 04:59:32.626456 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:32.626464 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:32.628342 (+ 1878us) server_negotiation.cc:407] Connection header received
0925 04:59:32.630162 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:32.630166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:32.630233 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:32.630329 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:32.631315 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:32.632108 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:32.633576 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:32.633976 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:32.634067 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:32.635313 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:32.635318 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:32.635322 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:59:32.635340 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:32.635373 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:59:32.635385 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:32.635455 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:32.635724 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 04:59:32.635854 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 04:59:32.637161 25536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:32.626976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:45411 (local address 127.6.29.131:49533)
0925 04:59:32.628177 (+ 1201us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:32.628221 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:32.629915 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:32.630500 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:32.630508 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:32.630606 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:59:32.631165 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:32.631187 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:32.632233 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:32.632237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:32.633459 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:32.633470 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:32.634160 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:32.634165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:32.634362 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:32.635009 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:59:32.635012 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:59:32.635060 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:59:32.635136 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:59:32.635508 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:59:32.635517 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:59:32.635524 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 04:59:32.635706 (+ 182us) client_negotiation.cc:240] Negotiation successful
0925 04:59:32.636078 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":376,"threads_started":1}
I20260925 04:59:32.638659 25534 heartbeater.cc:347] Connected to a master server at 127.6.29.190:45411
I20260925 04:59:32.638926 25534 heartbeater.cc:464] Registering TS with master...
I20260925 04:59:32.639819 25534 heartbeater.cc:511] Master 127.6.29.190:45411 requested a full tablet report, sending...
I20260925 04:59:32.641273 25087 ts_manager.cc:194] Registered new tserver with Master: 6c106d1c02a341daa2c7f23166afd23e (127.6.29.131:39117)
I20260925 04:59:32.642036 25087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.29.131:49533
I20260925 04:59:32.653389 6262 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:59:32.663738 25275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:32.657416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42584 (local address 127.6.29.190:45411)
0925 04:59:32.657648 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:32.657657 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:32.658044 (+ 387us) server_negotiation.cc:407] Connection header received
0925 04:59:32.658248 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:32.658252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:32.658326 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:32.658428 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:32.658906 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:32.659679 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:32.661177 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:32.661737 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:32.661874 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:32.663029 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:32.663033 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:32.663037 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:59:32.663055 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:32.663085 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:59:32.663096 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:32.663251 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:32.663433 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 04:59:32.663489 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 04:59:32.667456 25087 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:59:32.674510 25087 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:59:32.704731 25548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:32.686880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:34567 (local address 127.0.0.1:33248)
0925 04:59:32.692884 (+ 6004us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:32.692939 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:32.693282 (+ 343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:32.696969 (+ 3687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:32.696980 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:32.697021 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:59:32.697538 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:32.697557 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:32.699333 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:32.699339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:32.701814 (+ 2475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:32.701829 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:32.703698 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:32.703704 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:32.703919 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:32.703980 (+ 61us) client_negotiation.cc:769] Sending connection context
0925 04:59:32.704141 (+ 161us) client_negotiation.cc:240] Negotiation successful
0925 04:59:32.704345 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5656,"mutex_wait_us":13,"thread_start_us":9435,"threads_started":1}
I20260925 04:59:32.704733 25547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:32.687434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33248 (local address 127.6.29.129:34567)
0925 04:59:32.696285 (+ 8851us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:32.696297 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:32.696333 (+ 36us) server_negotiation.cc:407] Connection header received
0925 04:59:32.696446 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:32.696455 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:32.696722 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:32.696914 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:59:32.697779 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:32.699196 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:32.702056 (+ 2860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:32.703533 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:32.703674 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:32.703985 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:32.704189 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 04:59:32.704353 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8657,"thread_start_us":8548,"threads_started":1}
I20260925 04:59:32.706395 25550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:32.689797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:46155 (local address 127.0.0.1:36610)
0925 04:59:32.697534 (+ 7737us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:32.697557 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:32.697701 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:32.698666 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:32.698674 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:32.698698 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:59:32.699285 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:32.699299 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:32.701182 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:32.701188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:32.703403 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:32.703419 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:32.705502 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:32.705507 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:32.705772 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:32.705806 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 04:59:32.705892 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 04:59:32.706028 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7604,"mutex_wait_us":1,"spinlock_wait_cycles":4352,"thread_start_us":7674,"threads_started":1}
I20260925 04:59:32.706532 25551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:32.690350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36610 (local address 127.6.29.130:46155)
0925 04:59:32.697534 (+ 7184us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:32.697543 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:32.697845 (+ 302us) server_negotiation.cc:407] Connection header received
0925 04:59:32.697953 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:32.697960 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:32.698201 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:32.698393 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:59:32.699451 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:32.701000 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:32.703608 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:32.705038 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:32.705155 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:32.705360 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:32.705997 (+ 637us) server_negotiation.cc:299] Negotiation successful
0925 04:59:32.706159 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7026,"thread_start_us":6904,"threads_started":1}
I20260925 04:59:32.707010 25546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:32.687723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.131:39117 (local address 127.0.0.1:39792)
0925 04:59:32.696581 (+ 8858us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:32.696599 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:32.696725 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:32.698833 (+ 2108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:32.698839 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:32.698866 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:59:32.699298 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:32.699307 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:32.700806 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:32.700812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:32.702726 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:32.702738 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:32.705450 (+ 2712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:32.705457 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:32.705656 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:32.705788 (+ 132us) client_negotiation.cc:769] Sending connection context
0925 04:59:32.705877 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 04:59:32.706009 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8724,"mutex_wait_us":93,"thread_start_us":4579,"threads_started":1}
I20260925 04:59:32.707876 25549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:32.688538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39792 (local address 127.6.29.131:39117)
0925 04:59:32.698012 (+ 9474us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:32.698021 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:32.698067 (+ 46us) server_negotiation.cc:407] Connection header received
0925 04:59:32.698150 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:32.698156 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:32.698396 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:32.698570 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:59:32.699526 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:32.700581 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:32.702908 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:32.704033 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:32.704168 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:32.705704 (+ 1536us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:32.707296 (+ 1592us) server_negotiation.cc:299] Negotiation successful
0925 04:59:32.707478 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9305,"thread_start_us":10193,"threads_started":1}
I20260925 04:59:32.709122 25208 tablet_service.cc:1467] Processing CreateTablet for tablet ab5eef2b40774ba0a72d19c90b0e4e63 (DEFAULT_TABLE table=test-table [id=41fdd43c738245ea82876c1ac4cb0a2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:59:32.710738 25208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab5eef2b40774ba0a72d19c90b0e4e63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:59:32.710276 25339 tablet_service.cc:1467] Processing CreateTablet for tablet ab5eef2b40774ba0a72d19c90b0e4e63 (DEFAULT_TABLE table=test-table [id=41fdd43c738245ea82876c1ac4cb0a2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:59:32.711095 25469 tablet_service.cc:1467] Processing CreateTablet for tablet ab5eef2b40774ba0a72d19c90b0e4e63 (DEFAULT_TABLE table=test-table [id=41fdd43c738245ea82876c1ac4cb0a2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:59:32.711665 25339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab5eef2b40774ba0a72d19c90b0e4e63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:59:32.712391 25469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab5eef2b40774ba0a72d19c90b0e4e63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:59:32.712540 25273 heartbeater.cc:503] Master 127.6.29.190:45411 was elected leader, sending a full tablet report...
I20260925 04:59:32.724560 25552 tablet_bootstrap.cc:492] T ab5eef2b40774ba0a72d19c90b0e4e63 P 6c106d1c02a341daa2c7f23166afd23e: Bootstrap starting.
I20260925 04:59:32.726763 25554 tablet_bootstrap.cc:492] T ab5eef2b40774ba0a72d19c90b0e4e63 P 69f7962ad09c412fbbfbec325c8a7bdb: Bootstrap starting.
I20260925 04:59:32.729954 25552 tablet_bootstrap.cc:654] T ab5eef2b40774ba0a72d19c90b0e4e63 P 6c106d1c02a341daa2c7f23166afd23e: Neither blocks nor log segments found. Creating new log.
I20260925 04:59:32.730530 25554 tablet_bootstrap.cc:654] T ab5eef2b40774ba0a72d19c90b0e4e63 P 69f7962ad09c412fbbfbec325c8a7bdb: Neither blocks nor log segments found. Creating new log.
I20260925 04:59:32.731303 25552 log.cc:824] T ab5eef2b40774ba0a72d19c90b0e4e63 P 6c106d1c02a341daa2c7f23166afd23e: Log is configured to *not* fsync() on all Append() calls
I20260925 04:59:32.731745 25554 log.cc:824] T ab5eef2b40774ba0a72d19c90b0e4e63 P 69f7962ad09c412fbbfbec325c8a7bdb: Log is configured to *not* fsync() on all Append() calls
I20260925 04:59:32.731887 25553 tablet_bootstrap.cc:492] T ab5eef2b40774ba0a72d19c90b0e4e63 P 3acf6f27ec8b446c85747c62cadd6fd4: Bootstrap starting.
I20260925 04:59:32.735092 25552 tablet_bootstrap.cc:492] T ab5eef2b40774ba0a72d19c90b0e4e63 P 6c106d1c02a341daa2c7f23166afd23e: No bootstrap required, opened a new log
I20260925 04:59:32.735396 25552 ts_tablet_manager.cc:1397] T ab5eef2b40774ba0a72d19c90b0e4e63 P 6c106d1c02a341daa2c7f23166afd23e: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.003s
I20260925 04:59:32.735596 25553 tablet_bootstrap.cc:654] T ab5eef2b40774ba0a72d19c90b0e4e63 P 3acf6f27ec8b446c85747c62cadd6fd4: Neither blocks nor log segments found. Creating new log.
I20260925 04:59:32.735656 25554 tablet_bootstrap.cc:492] T ab5eef2b40774ba0a72d19c90b0e4e63 P 69f7962ad09c412fbbfbec325c8a7bdb: No bootstrap required, opened a new log
I20260925 04:59:32.735944 25554 ts_tablet_manager.cc:1397] T ab5eef2b40774ba0a72d19c90b0e4e63 P 69f7962ad09c412fbbfbec325c8a7bdb: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.004s
I20260925 04:59:32.736856 25553 log.cc:824] T ab5eef2b40774ba0a72d19c90b0e4e63 P 3acf6f27ec8b446c85747c62cadd6fd4: Log is configured to *not* fsync() on all Append() calls
I20260925 04:59:32.740911 25553 tablet_bootstrap.cc:492] T ab5eef2b40774ba0a72d19c90b0e4e63 P 3acf6f27ec8b446c85747c62cadd6fd4: No bootstrap required, opened a new log
I20260925 04:59:32.741178 25553 ts_tablet_manager.cc:1397] T ab5eef2b40774ba0a72d19c90b0e4e63 P 3acf6f27ec8b446c85747c62cadd6fd4: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260925 04:59:32.743942 25554 raft_consensus.cc:348] T ab5eef2b40774ba0a72d19c90b0e4e63 P 69f7962ad09c412fbbfbec325c8a7bdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3acf6f27ec8b446c85747c62cadd6fd4" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 34567 } } peers { permanent_uuid: "69f7962ad09c412fbbfbec325c8a7bdb" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 46155 } } peers { permanent_uuid: "6c106d1c02a341daa2c7f23166afd23e" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 39117 } }
I20260925 04:59:32.744324 25554 raft_consensus.cc:374] T ab5eef2b40774ba0a72d19c90b0e4e63 P 69f7962ad09c412fbbfbec325c8a7bdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:59:32.744431 25554 raft_consensus.cc:729] T ab5eef2b40774ba0a72d19c90b0e4e63 P 69f7962ad09c412fbbfbec325c8a7bdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69f7962ad09c412fbbfbec325c8a7bdb, State: Initialized, Role: FOLLOWER
I20260925 04:59:32.745023 25554 consensus_queue.cc:260] T ab5eef2b40774ba0a72d19c90b0e4e63 P 69f7962ad09c412fbbfbec325c8a7bdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3acf6f27ec8b446c85747c62cadd6fd4" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 34567 } } peers { permanent_uuid: "69f7962ad09c412fbbfbec325c8a7bdb" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 46155 } } peers { permanent_uuid: "6c106d1c02a341daa2c7f23166afd23e" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 39117 } }
I20260925 04:59:32.746165 25404 heartbeater.cc:503] Master 127.6.29.190:45411 was elected leader, sending a full tablet report...
I20260925 04:59:32.746261 25552 raft_consensus.cc:348] T ab5eef2b40774ba0a72d19c90b0e4e63 P 6c106d1c02a341daa2c7f23166afd23e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3acf6f27ec8b446c85747c62cadd6fd4" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 34567 } } peers { permanent_uuid: "69f7962ad09c412fbbfbec325c8a7bdb" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 46155 } } peers { permanent_uuid: "6c106d1c02a341daa2c7f23166afd23e" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 39117 } }
I20260925 04:59:32.746686 25552 raft_consensus.cc:374] T ab5eef2b40774ba0a72d19c90b0e4e63 P 6c106d1c02a341daa2c7f23166afd23e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:59:32.746701 25554 ts_tablet_manager.cc:1428] T ab5eef2b40774ba0a72d19c90b0e4e63 P 69f7962ad09c412fbbfbec325c8a7bdb: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260925 04:59:32.746788 25552 raft_consensus.cc:729] T ab5eef2b40774ba0a72d19c90b0e4e63 P 6c106d1c02a341daa2c7f23166afd23e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c106d1c02a341daa2c7f23166afd23e, State: Initialized, Role: FOLLOWER
I20260925 04:59:32.748377 25552 consensus_queue.cc:260] T ab5eef2b40774ba0a72d19c90b0e4e63 P 6c106d1c02a341daa2c7f23166afd23e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3acf6f27ec8b446c85747c62cadd6fd4" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 34567 } } peers { permanent_uuid: "69f7962ad09c412fbbfbec325c8a7bdb" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 46155 } } peers { permanent_uuid: "6c106d1c02a341daa2c7f23166afd23e" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 39117 } }
I20260925 04:59:32.748800 25559 raft_consensus.cc:482] T ab5eef2b40774ba0a72d19c90b0e4e63 P 6c106d1c02a341daa2c7f23166afd23e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:59:32.749110 25559 raft_consensus.cc:504] T ab5eef2b40774ba0a72d19c90b0e4e63 P 6c106d1c02a341daa2c7f23166afd23e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3acf6f27ec8b446c85747c62cadd6fd4" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 34567 } } peers { permanent_uuid: "69f7962ad09c412fbbfbec325c8a7bdb" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 46155 } } peers { permanent_uuid: "6c106d1c02a341daa2c7f23166afd23e" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 39117 } }
I20260925 04:59:32.755908 25559 leader_election.cc:288] T ab5eef2b40774ba0a72d19c90b0e4e63 P 6c106d1c02a341daa2c7f23166afd23e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3acf6f27ec8b446c85747c62cadd6fd4 (127.6.29.129:34567), 69f7962ad09c412fbbfbec325c8a7bdb (127.6.29.130:46155)
I20260925 04:59:32.756649 25534 heartbeater.cc:503] Master 127.6.29.190:45411 was elected leader, sending a full tablet report...
I20260925 04:59:32.761318 25536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:32.751358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:34567 (local address 127.6.29.131:59667)
0925 04:59:32.751746 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:32.751791 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:32.751905 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:32.753226 (+ 1321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:32.753235 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:32.753277 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:59:32.753539 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:32.753555 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:32.755361 (+ 1806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:32.755370 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:32.757808 (+ 2438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:32.757824 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:32.760158 (+ 2334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:32.760168 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:32.760472 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:32.760526 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 04:59:32.760654 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 04:59:32.760863 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273}
I20260925 04:59:32.762307 25547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:32.751829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:59667 (local address 127.6.29.129:34567)
0925 04:59:32.752107 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:32.752116 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:32.752142 (+ 26us) server_negotiation.cc:407] Connection header received
0925 04:59:32.752233 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:32.752242 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:32.752355 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:32.752478 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:59:32.753768 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:32.755180 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:32.758113 (+ 2933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:32.759721 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:32.759939 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:32.760048 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:32.761771 (+ 1723us) server_negotiation.cc:299] Negotiation successful
0925 04:59:32.761975 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 04:59:32.763535 25228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab5eef2b40774ba0a72d19c90b0e4e63" candidate_uuid: "6c106d1c02a341daa2c7f23166afd23e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3acf6f27ec8b446c85747c62cadd6fd4" is_pre_election: true
W20260925 04:59:32.766739 25425 leader_election.cc:341] T ab5eef2b40774ba0a72d19c90b0e4e63 P 6c106d1c02a341daa2c7f23166afd23e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3acf6f27ec8b446c85747c62cadd6fd4 (127.6.29.129:34567): Illegal state: must be running to vote when last-logged opid is not known
I20260925 04:59:32.769567 25553 raft_consensus.cc:348] T ab5eef2b40774ba0a72d19c90b0e4e63 P 3acf6f27ec8b446c85747c62cadd6fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3acf6f27ec8b446c85747c62cadd6fd4" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 34567 } } peers { permanent_uuid: "69f7962ad09c412fbbfbec325c8a7bdb" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 46155 } } peers { permanent_uuid: "6c106d1c02a341daa2c7f23166afd23e" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 39117 } }
I20260925 04:59:32.770265 25553 raft_consensus.cc:374] T ab5eef2b40774ba0a72d19c90b0e4e63 P 3acf6f27ec8b446c85747c62cadd6fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:59:32.770443 25553 raft_consensus.cc:729] T ab5eef2b40774ba0a72d19c90b0e4e63 P 3acf6f27ec8b446c85747c62cadd6fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3acf6f27ec8b446c85747c62cadd6fd4, State: Initialized, Role: FOLLOWER
I20260925 04:59:32.771797 25553 consensus_queue.cc:260] T ab5eef2b40774ba0a72d19c90b0e4e63 P 3acf6f27ec8b446c85747c62cadd6fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3acf6f27ec8b446c85747c62cadd6fd4" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 34567 } } peers { permanent_uuid: "69f7962ad09c412fbbfbec325c8a7bdb" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 46155 } } peers { permanent_uuid: "6c106d1c02a341daa2c7f23166afd23e" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 39117 } }
I20260925 04:59:32.773047 25536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:32.752540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:46155 (local address 127.6.29.131:48243)
0925 04:59:32.763092 (+ 10552us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:32.763117 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:32.763338 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:32.764009 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:32.764016 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:32.764051 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:59:32.764315 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:32.764330 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:32.767137 (+ 2807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:32.767143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:32.769804 (+ 2661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:32.769824 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:32.772261 (+ 2437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:32.772267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:32.772468 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:32.772499 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 04:59:32.772584 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 04:59:32.772718 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10397,"thread_start_us":18814,"threads_started":1}
I20260925 04:59:32.773265 25551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:32.752876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:48243 (local address 127.6.29.130:46155)
0925 04:59:32.753102 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:32.753110 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:32.763208 (+ 10098us) server_negotiation.cc:407] Connection header received
0925 04:59:32.763523 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:32.763532 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:32.763661 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:32.763741 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:59:32.764623 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:32.766016 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:32.770044 (+ 4028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:32.771729 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:32.771907 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:32.771988 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:32.772725 (+ 737us) server_negotiation.cc:299] Negotiation successful
0925 04:59:32.772927 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 04:59:32.773383 25552 ts_tablet_manager.cc:1428] T ab5eef2b40774ba0a72d19c90b0e4e63 P 6c106d1c02a341daa2c7f23166afd23e: Time spent starting tablet: real 0.038s user 0.016s sys 0.021s
I20260925 04:59:32.774322 25359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab5eef2b40774ba0a72d19c90b0e4e63" candidate_uuid: "6c106d1c02a341daa2c7f23166afd23e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f7962ad09c412fbbfbec325c8a7bdb" is_pre_election: true
I20260925 04:59:32.774768 25553 ts_tablet_manager.cc:1428] T ab5eef2b40774ba0a72d19c90b0e4e63 P 3acf6f27ec8b446c85747c62cadd6fd4: Time spent starting tablet: real 0.033s user 0.009s sys 0.008s
I20260925 04:59:32.774878 25359 raft_consensus.cc:2454] T ab5eef2b40774ba0a72d19c90b0e4e63 P 69f7962ad09c412fbbfbec325c8a7bdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c106d1c02a341daa2c7f23166afd23e in term 0.
I20260925 04:59:32.775475 25422 leader_election.cc:302] T ab5eef2b40774ba0a72d19c90b0e4e63 P 6c106d1c02a341daa2c7f23166afd23e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 69f7962ad09c412fbbfbec325c8a7bdb, 6c106d1c02a341daa2c7f23166afd23e; no voters: 3acf6f27ec8b446c85747c62cadd6fd4
I20260925 04:59:32.775918 25560 raft_consensus.cc:2781] T ab5eef2b40774ba0a72d19c90b0e4e63 P 6c106d1c02a341daa2c7f23166afd23e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:59:32.776077 25560 raft_consensus.cc:482] T ab5eef2b40774ba0a72d19c90b0e4e63 P 6c106d1c02a341daa2c7f23166afd23e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:59:32.776214 25560 raft_consensus.cc:3037] T ab5eef2b40774ba0a72d19c90b0e4e63 P 6c106d1c02a341daa2c7f23166afd23e [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:59:32.778915 25560 raft_consensus.cc:504] T ab5eef2b40774ba0a72d19c90b0e4e63 P 6c106d1c02a341daa2c7f23166afd23e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3acf6f27ec8b446c85747c62cadd6fd4" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 34567 } } peers { permanent_uuid: "69f7962ad09c412fbbfbec325c8a7bdb" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 46155 } } peers { permanent_uuid: "6c106d1c02a341daa2c7f23166afd23e" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 39117 } }
I20260925 04:59:32.779552 25560 leader_election.cc:288] T ab5eef2b40774ba0a72d19c90b0e4e63 P 6c106d1c02a341daa2c7f23166afd23e [CANDIDATE]: Term 1 election: Requested vote from peers 3acf6f27ec8b446c85747c62cadd6fd4 (127.6.29.129:34567), 69f7962ad09c412fbbfbec325c8a7bdb (127.6.29.130:46155)
I20260925 04:59:32.780043 25359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab5eef2b40774ba0a72d19c90b0e4e63" candidate_uuid: "6c106d1c02a341daa2c7f23166afd23e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69f7962ad09c412fbbfbec325c8a7bdb"
I20260925 04:59:32.780058 25228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab5eef2b40774ba0a72d19c90b0e4e63" candidate_uuid: "6c106d1c02a341daa2c7f23166afd23e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3acf6f27ec8b446c85747c62cadd6fd4"
I20260925 04:59:32.780253 25359 raft_consensus.cc:3037] T ab5eef2b40774ba0a72d19c90b0e4e63 P 69f7962ad09c412fbbfbec325c8a7bdb [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:59:32.780279 25228 raft_consensus.cc:3037] T ab5eef2b40774ba0a72d19c90b0e4e63 P 3acf6f27ec8b446c85747c62cadd6fd4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:59:32.782953 25359 raft_consensus.cc:2454] T ab5eef2b40774ba0a72d19c90b0e4e63 P 69f7962ad09c412fbbfbec325c8a7bdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c106d1c02a341daa2c7f23166afd23e in term 1.
I20260925 04:59:32.782954 25228 raft_consensus.cc:2454] T ab5eef2b40774ba0a72d19c90b0e4e63 P 3acf6f27ec8b446c85747c62cadd6fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c106d1c02a341daa2c7f23166afd23e in term 1.
I20260925 04:59:32.783353 25425 leader_election.cc:302] T ab5eef2b40774ba0a72d19c90b0e4e63 P 6c106d1c02a341daa2c7f23166afd23e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3acf6f27ec8b446c85747c62cadd6fd4, 6c106d1c02a341daa2c7f23166afd23e; no voters:
I20260925 04:59:32.783602 25560 raft_consensus.cc:2781] T ab5eef2b40774ba0a72d19c90b0e4e63 P 6c106d1c02a341daa2c7f23166afd23e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:59:32.783795 25560 raft_consensus.cc:686] T ab5eef2b40774ba0a72d19c90b0e4e63 P 6c106d1c02a341daa2c7f23166afd23e [term 1 LEADER]: Becoming Leader. State: Replica: 6c106d1c02a341daa2c7f23166afd23e, State: Running, Role: LEADER
I20260925 04:59:32.784250 25560 consensus_queue.cc:237] T ab5eef2b40774ba0a72d19c90b0e4e63 P 6c106d1c02a341daa2c7f23166afd23e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3acf6f27ec8b446c85747c62cadd6fd4" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 34567 } } peers { permanent_uuid: "69f7962ad09c412fbbfbec325c8a7bdb" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 46155 } } peers { permanent_uuid: "6c106d1c02a341daa2c7f23166afd23e" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 39117 } }
I20260925 04:59:32.788466 25086 catalog_manager.cc:5495] T ab5eef2b40774ba0a72d19c90b0e4e63 P 6c106d1c02a341daa2c7f23166afd23e reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c106d1c02a341daa2c7f23166afd23e (127.6.29.131). New cstate: current_term: 1 leader_uuid: "6c106d1c02a341daa2c7f23166afd23e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3acf6f27ec8b446c85747c62cadd6fd4" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 34567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69f7962ad09c412fbbfbec325c8a7bdb" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 46155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c106d1c02a341daa2c7f23166afd23e" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 39117 } health_report { overall_health: HEALTHY } } }
I20260925 04:59:32.813582 25549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:32.808418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39808 (local address 127.6.29.131:39117)
0925 04:59:32.808731 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:32.808736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:32.808754 (+ 18us) server_negotiation.cc:407] Connection header received
0925 04:59:32.808833 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:32.808836 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:32.808911 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:32.809029 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:59:32.809331 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:32.810059 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:32.811387 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:32.811893 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:32.811986 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:32.812474 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:59:32.812980 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:59:32.813113 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:32.813252 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 04:59:32.813311 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197}
I20260925 04:59:32.821955 25228 raft_consensus.cc:1261] T ab5eef2b40774ba0a72d19c90b0e4e63 P 3acf6f27ec8b446c85747c62cadd6fd4 [term 1 FOLLOWER]: Refusing update from remote peer 6c106d1c02a341daa2c7f23166afd23e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:59:32.822008 25359 raft_consensus.cc:1261] T ab5eef2b40774ba0a72d19c90b0e4e63 P 69f7962ad09c412fbbfbec325c8a7bdb [term 1 FOLLOWER]: Refusing update from remote peer 6c106d1c02a341daa2c7f23166afd23e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:59:32.822897 25560 consensus_queue.cc:1035] T ab5eef2b40774ba0a72d19c90b0e4e63 P 6c106d1c02a341daa2c7f23166afd23e [LEADER]: Connected to new peer: Peer: permanent_uuid: "69f7962ad09c412fbbfbec325c8a7bdb" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 46155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:59:32.823148 25559 consensus_queue.cc:1035] T ab5eef2b40774ba0a72d19c90b0e4e63 P 6c106d1c02a341daa2c7f23166afd23e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3acf6f27ec8b446c85747c62cadd6fd4" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 34567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:59:32.841970 25566 mvcc.cc:205] Tried to move back new op lower bound from 7333119479067934720 to 7333119478928420864. Current Snapshot: MvccSnapshot[applied={T|T < 7333119479067934720}]
I20260925 04:59:32.843407 25567 mvcc.cc:205] Tried to move back new op lower bound from 7333119479067934720 to 7333119478928420864. Current Snapshot: MvccSnapshot[applied={T|T < 7333119479067934720}]
I20260925 04:59:32.856626 25087 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:59:32.856923 25087 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:59:32.860718 25087 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bae37a2535d640eea6da537f180e51ad: Sending DeleteTablet for 3 replicas of tablet ab5eef2b40774ba0a72d19c90b0e4e63
I20260925 04:59:32.861913 25208 tablet_service.cc:1514] Processing DeleteTablet for tablet ab5eef2b40774ba0a72d19c90b0e4e63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:59:32 UTC) from {username='slave'} at 127.0.0.1:33248
I20260925 04:59:32.862354 25339 tablet_service.cc:1514] Processing DeleteTablet for tablet ab5eef2b40774ba0a72d19c90b0e4e63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:59:32 UTC) from {username='slave'} at 127.0.0.1:36610
I20260925 04:59:32.862936 25469 tablet_service.cc:1514] Processing DeleteTablet for tablet ab5eef2b40774ba0a72d19c90b0e4e63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:59:32 UTC) from {username='slave'} at 127.0.0.1:39792
I20260925 04:59:32.863186 25575 tablet_replica.cc:330] T ab5eef2b40774ba0a72d19c90b0e4e63 P 3acf6f27ec8b446c85747c62cadd6fd4: stopping tablet replica
I20260925 04:59:32.863793 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 25146
I20260925 04:59:32.863754 25575 raft_consensus.cc:2229] T ab5eef2b40774ba0a72d19c90b0e4e63 P 3acf6f27ec8b446c85747c62cadd6fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:59:32.864645 25575 raft_consensus.cc:2258] T ab5eef2b40774ba0a72d19c90b0e4e63 P 3acf6f27ec8b446c85747c62cadd6fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:59:32.867127 25575 ts_tablet_manager.cc:1905] T ab5eef2b40774ba0a72d19c90b0e4e63 P 3acf6f27ec8b446c85747c62cadd6fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:59:32.876926 25575 ts_tablet_manager.cc:1918] T ab5eef2b40774ba0a72d19c90b0e4e63 P 3acf6f27ec8b446c85747c62cadd6fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:59:32.877230 25575 log.cc:1196] T ab5eef2b40774ba0a72d19c90b0e4e63 P 3acf6f27ec8b446c85747c62cadd6fd4: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-0/wal/wals/ab5eef2b40774ba0a72d19c90b0e4e63
I20260925 04:59:32.877867 25575 ts_tablet_manager.cc:1939] T ab5eef2b40774ba0a72d19c90b0e4e63 P 3acf6f27ec8b446c85747c62cadd6fd4: Deleting consensus metadata
I20260925 04:59:32.880012 25074 catalog_manager.cc:4841] TS 3acf6f27ec8b446c85747c62cadd6fd4 (127.6.29.129:34567): tablet ab5eef2b40774ba0a72d19c90b0e4e63 (table test-table [id=41fdd43c738245ea82876c1ac4cb0a2d]) successfully deleted
I20260925 04:59:32.880784 25577 tablet_replica.cc:330] T ab5eef2b40774ba0a72d19c90b0e4e63 P 6c106d1c02a341daa2c7f23166afd23e: stopping tablet replica
I20260925 04:59:32.881160 25577 raft_consensus.cc:2229] T ab5eef2b40774ba0a72d19c90b0e4e63 P 6c106d1c02a341daa2c7f23166afd23e [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:59:32.881698 25577 raft_consensus.cc:2258] T ab5eef2b40774ba0a72d19c90b0e4e63 P 6c106d1c02a341daa2c7f23166afd23e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:59:32.882859 25547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:32.866004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33264 (local address 127.6.29.129:34567)
0925 04:59:32.866246 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:32.866255 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:32.870292 (+ 4037us) server_negotiation.cc:407] Connection header received
0925 04:59:32.870536 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:32.870545 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:32.870679 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:32.870806 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:32.871678 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:32.873275 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:32.877545 (+ 4270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:32.878518 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:32.878646 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:32.880561 (+ 1915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:32.880574 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:32.880597 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:59:32.881582 (+ 985us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:32.881653 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:59:32.881685 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:32.881797 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:32.882144 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 04:59:32.882306 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 04:59:32.883554 25577 ts_tablet_manager.cc:1905] T ab5eef2b40774ba0a72d19c90b0e4e63 P 6c106d1c02a341daa2c7f23166afd23e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:59:32.884268 25268 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:59:32.890115 25577 ts_tablet_manager.cc:1918] T ab5eef2b40774ba0a72d19c90b0e4e63 P 6c106d1c02a341daa2c7f23166afd23e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:59:32.890277 25577 log.cc:1196] T ab5eef2b40774ba0a72d19c90b0e4e63 P 6c106d1c02a341daa2c7f23166afd23e: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-2/wal/wals/ab5eef2b40774ba0a72d19c90b0e4e63
I20260925 04:59:32.890758 25577 ts_tablet_manager.cc:1939] T ab5eef2b40774ba0a72d19c90b0e4e63 P 6c106d1c02a341daa2c7f23166afd23e: Deleting consensus metadata
I20260925 04:59:32.900349 25576 tablet_replica.cc:330] T ab5eef2b40774ba0a72d19c90b0e4e63 P 69f7962ad09c412fbbfbec325c8a7bdb: stopping tablet replica
I20260925 04:59:32.900691 25073 catalog_manager.cc:4841] TS 6c106d1c02a341daa2c7f23166afd23e (127.6.29.131:39117): tablet ab5eef2b40774ba0a72d19c90b0e4e63 (table test-table [id=41fdd43c738245ea82876c1ac4cb0a2d]) successfully deleted
I20260925 04:59:32.900859 25576 raft_consensus.cc:2229] T ab5eef2b40774ba0a72d19c90b0e4e63 P 69f7962ad09c412fbbfbec325c8a7bdb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:59:32.901407 25576 raft_consensus.cc:2258] T ab5eef2b40774ba0a72d19c90b0e4e63 P 69f7962ad09c412fbbfbec325c8a7bdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:59:32.903800 25576 ts_tablet_manager.cc:1905] T ab5eef2b40774ba0a72d19c90b0e4e63 P 69f7962ad09c412fbbfbec325c8a7bdb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:59:32.910880 25576 ts_tablet_manager.cc:1918] T ab5eef2b40774ba0a72d19c90b0e4e63 P 69f7962ad09c412fbbfbec325c8a7bdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:59:32.911051 25576 log.cc:1196] T ab5eef2b40774ba0a72d19c90b0e4e63 P 69f7962ad09c412fbbfbec325c8a7bdb: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790312053666503-6262-0/minicluster-data/ts-1/wal/wals/ab5eef2b40774ba0a72d19c90b0e4e63
I20260925 04:59:32.911592 25576 ts_tablet_manager.cc:1939] T ab5eef2b40774ba0a72d19c90b0e4e63 P 69f7962ad09c412fbbfbec325c8a7bdb: Deleting consensus metadata
I20260925 04:59:32.913558 25071 catalog_manager.cc:4841] TS 69f7962ad09c412fbbfbec325c8a7bdb (127.6.29.130:46155): tablet ab5eef2b40774ba0a72d19c90b0e4e63 (table test-table [id=41fdd43c738245ea82876c1ac4cb0a2d]) successfully deleted
I20260925 04:59:32.981041 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 25146
I20260925 04:59:33.006055 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 25277
I20260925 04:59:33.013136 25551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:33.006885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36618 (local address 127.6.29.130:46155)
0925 04:59:33.007083 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:33.007089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:33.007110 (+ 21us) server_negotiation.cc:407] Connection header received
0925 04:59:33.007352 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:33.007358 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:33.007471 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:33.007580 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:33.008024 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:33.008763 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:33.009988 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:33.010410 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:33.010543 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:33.011643 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:33.011652 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:33.011666 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:59:33.012406 (+ 740us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:33.012454 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:59:33.012474 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:33.012612 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:33.012701 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 04:59:33.012808 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 04:59:33.013855 25399 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:59:33.103948 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 25277
I20260925 04:59:33.121239 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 25407
I20260925 04:59:33.129418 25549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:33.122057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39822 (local address 127.6.29.131:39117)
0925 04:59:33.122253 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:33.122259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:33.122281 (+ 22us) server_negotiation.cc:407] Connection header received
0925 04:59:33.122448 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:33.122454 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:33.122612 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:33.122750 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:33.123173 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:33.124389 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:33.125763 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:33.126343 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:33.126476 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:33.127632 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:33.127645 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:33.127661 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:59:33.128585 (+ 924us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:33.128655 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:59:33.128681 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:33.128798 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:33.128944 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 04:59:33.129054 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 04:59:33.130271 25529 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:59:33.250679 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 25407
I20260925 04:59:33.269908 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 25056
I20260925 04:59:33.271011 25117 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:59:33.380759 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 25056
2026-09-25T04:59:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2779 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:59:33 dist-test-slave-xzjr krb5kdc[25585](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:59:33 dist-test-slave-xzjr krb5kdc[25585](info): set up 2 sockets
Sep 25 04:59:33 dist-test-slave-xzjr krb5kdc[25585](info): commencing operation
krb5kdc: starting...
W20260925 04:59:34.496467 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 04:59:34 dist-test-slave-xzjr krb5kdc[25585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:59:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:59:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.29.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
I20260925 04:59:34.760668 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:43545
--webserver_interface=127.6.29.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:33537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:43545
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:59:35.097793 25598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:59:35.098119 25598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:59:35.098306 25598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:59:35.105638 25598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:59:35.105753 25598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:59:35.105866 25598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:59:35.105971 25598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:59:35.106043 25598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:59:35.116241 25598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:33537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:43545
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:43545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:59:35.117667 25598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:59:35.119026 25598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:59:35.131309 25606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:35.131428 25603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:35.131407 25604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:59:35.132414 25598 server_base.cc:1034] running on GCE node
I20260925 04:59:35.133251 25598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:59:35.134706 25598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:59:35.135869 25598 hybrid_clock.cc:648] HybridClock initialized: now 1790312375135825 us; error 36 us; skew 500 ppm
I20260925 04:59:35.136189 25598 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:59:35 dist-test-slave-xzjr krb5kdc[25585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312375, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:59:35.143318 25598 init.cc:377] Logged in from keytab as kudu/127.6.29.190@KRBTEST.COM (short username kudu)
I20260925 04:59:35.144858 25598 webserver.cc:460] Webserver started at http://127.6.29.190:39157/ using document root <none> and password file <none>
I20260925 04:59:35.145519 25598 fs_manager.cc:359] Metadata directory not provided
I20260925 04:59:35.145627 25598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:59:35.145888 25598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:59:35.149040 25598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "80f9a92059b445608f2f097541ab8016"
format_stamp: "Formatted at 2026-09-25 04:59:35 on dist-test-slave-xzjr"
server_key: "42136bd6da4b240efd7a7b26f198507a"
server_key_iv: "959103bad5c79475010ca6f594286e8a"
server_key_version: "encryptionkey@0"
I20260925 04:59:35.149737 25598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "80f9a92059b445608f2f097541ab8016"
format_stamp: "Formatted at 2026-09-25 04:59:35 on dist-test-slave-xzjr"
server_key: "42136bd6da4b240efd7a7b26f198507a"
server_key_iv: "959103bad5c79475010ca6f594286e8a"
server_key_version: "encryptionkey@0"
I20260925 04:59:35.155025 25598 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 04:59:35.158910 25613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:35.160444 25598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:59:35.160641 25598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "80f9a92059b445608f2f097541ab8016"
format_stamp: "Formatted at 2026-09-25 04:59:35 on dist-test-slave-xzjr"
server_key: "42136bd6da4b240efd7a7b26f198507a"
server_key_iv: "959103bad5c79475010ca6f594286e8a"
server_key_version: "encryptionkey@0"
I20260925 04:59:35.160861 25598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:59:35.178555 25598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:59:35.183223 25598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:59:35.183542 25598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:59:35.202090 25598 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:43545
I20260925 04:59:35.202118 25664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:43545 every 8 connection(s)
I20260925 04:59:35.203576 25598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:59:35.207587 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 25598
I20260925 04:59:35.207836 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 04:59:35.208009 25665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:59:35.208384 6262 external_mini_cluster.cc:1428] Setting key 683941fcf0610e24d750510cdbb27a50
I20260925 04:59:35.223404 25665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80f9a92059b445608f2f097541ab8016: Bootstrap starting.
Sep 25 04:59:35 dist-test-slave-xzjr krb5kdc[25585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:59:35.227180 25665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80f9a92059b445608f2f097541ab8016: Neither blocks nor log segments found. Creating new log.
I20260925 04:59:35.228336 25665 log.cc:824] T 00000000000000000000000000000000 P 80f9a92059b445608f2f097541ab8016: Log is configured to *not* fsync() on all Append() calls
I20260925 04:59:35.232012 25665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80f9a92059b445608f2f097541ab8016: No bootstrap required, opened a new log
I20260925 04:59:35.234251 25668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:35.210949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59358 (local address 127.6.29.190:43545)
0925 04:59:35.211909 (+ 960us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:35.211925 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:35.211978 (+ 53us) server_negotiation.cc:407] Connection header received
0925 04:59:35.213914 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:35.213949 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:35.214502 (+ 553us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:35.215000 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:35.216505 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:35.218454 (+ 1949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:35.219787 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:35.220934 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:35.221140 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:35.225683 (+ 4543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:35.225692 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:35.225707 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:59:35.225754 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:35.229224 (+ 3470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:35.230169 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:35.230178 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:35.230186 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:35.230416 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:35.230865 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:35.230870 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:35.230873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:35.231616 (+ 743us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:35.231950 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:35.232366 (+ 416us) server_negotiation.cc:299] Negotiation successful
0925 04:59:35.232865 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":427,"threads_started":1}
I20260925 04:59:35.240967 25665 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80f9a92059b445608f2f097541ab8016 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80f9a92059b445608f2f097541ab8016" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43545 } }
I20260925 04:59:35.241343 25665 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80f9a92059b445608f2f097541ab8016 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:59:35.241420 25665 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80f9a92059b445608f2f097541ab8016 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80f9a92059b445608f2f097541ab8016, State: Initialized, Role: FOLLOWER
I20260925 04:59:35.241940 25665 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80f9a92059b445608f2f097541ab8016 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80f9a92059b445608f2f097541ab8016" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43545 } }
I20260925 04:59:35.242210 25665 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80f9a92059b445608f2f097541ab8016 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:59:35.242367 25665 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80f9a92059b445608f2f097541ab8016 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:59:35.242570 25665 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80f9a92059b445608f2f097541ab8016 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:59:35.244911 25665 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80f9a92059b445608f2f097541ab8016 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80f9a92059b445608f2f097541ab8016" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43545 } }
I20260925 04:59:35.245435 25665 leader_election.cc:302] T 00000000000000000000000000000000 P 80f9a92059b445608f2f097541ab8016 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80f9a92059b445608f2f097541ab8016; no voters:
I20260925 04:59:35.246130 25665 leader_election.cc:288] T 00000000000000000000000000000000 P 80f9a92059b445608f2f097541ab8016 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:59:35.246286 25670 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80f9a92059b445608f2f097541ab8016 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:59:35.246992 25670 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80f9a92059b445608f2f097541ab8016 [term 1 LEADER]: Becoming Leader. State: Replica: 80f9a92059b445608f2f097541ab8016, State: Running, Role: LEADER
I20260925 04:59:35.247506 25670 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80f9a92059b445608f2f097541ab8016 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80f9a92059b445608f2f097541ab8016" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43545 } }
I20260925 04:59:35.248361 25665 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80f9a92059b445608f2f097541ab8016 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:59:35.250830 25672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80f9a92059b445608f2f097541ab8016 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80f9a92059b445608f2f097541ab8016. Latest consensus state: current_term: 1 leader_uuid: "80f9a92059b445608f2f097541ab8016" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80f9a92059b445608f2f097541ab8016" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43545 } } }
I20260925 04:59:35.250972 25671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80f9a92059b445608f2f097541ab8016 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80f9a92059b445608f2f097541ab8016" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80f9a92059b445608f2f097541ab8016" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43545 } } }
I20260925 04:59:35.251259 25672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80f9a92059b445608f2f097541ab8016 [sys.catalog]: This master's current role is: LEADER
I20260925 04:59:35.251504 25671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80f9a92059b445608f2f097541ab8016 [sys.catalog]: This master's current role is: LEADER
I20260925 04:59:35.253863 25677 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:59:35.259959 25677 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:59:35.270113 25677 catalog_manager.cc:1347] Generated new cluster ID: 5d1594feee2d4fffb2786d651223ecfb
I20260925 04:59:35.270288 25677 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:59:35.289749 25677 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:59:35.291143 25677 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:59:35.306393 25677 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80f9a92059b445608f2f097541ab8016: Generated new TSK 0
I20260925 04:59:35.307512 25677 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.29.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab.
Entry for principal kudu/127.6.29.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab.
I20260925 04:59:35.387250 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.129:0
--local_ip_for_outbound_sockets=127.6.29.129
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:43545
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:33537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:59:35.745507 25691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:59:35.745803 25691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:59:35.745992 25691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:59:35.753913 25691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:59:35.754061 25691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:59:35.754242 25691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.129
I20260925 04:59:35.764614 25691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:33537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.129.keytab
--principal=kudu/127.6.29.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:43545
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.129
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:59:35.766070 25691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:59:35.767434 25691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:59:35.781296 25699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:35.781404 25697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:35.781378 25696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:59:35.782240 25691 server_base.cc:1034] running on GCE node
I20260925 04:59:35.782871 25691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:59:35.783941 25691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:59:35.785182 25691 hybrid_clock.cc:648] HybridClock initialized: now 1790312375785129 us; error 46 us; skew 500 ppm
I20260925 04:59:35.785506 25691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:59:35 dist-test-slave-xzjr krb5kdc[25585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312375, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:59:35.788777 25691 init.cc:377] Logged in from keytab as kudu/127.6.29.129@KRBTEST.COM (short username kudu)
I20260925 04:59:35.790309 25691 webserver.cc:460] Webserver started at http://127.6.29.129:45123/ using document root <none> and password file <none>
I20260925 04:59:35.791016 25691 fs_manager.cc:359] Metadata directory not provided
I20260925 04:59:35.791172 25691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:59:35.791484 25691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:59:35.794448 25691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-0/data/instance:
uuid: "e6567b3b47a2425483a479140fdad1ef"
format_stamp: "Formatted at 2026-09-25 04:59:35 on dist-test-slave-xzjr"
server_key: "ca917d0e57cd5d85aa91ab5ac7fb6b10"
server_key_iv: "42505e15edd47800d2343bb72be1037d"
server_key_version: "encryptionkey@0"
I20260925 04:59:35.795249 25691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance:
uuid: "e6567b3b47a2425483a479140fdad1ef"
format_stamp: "Formatted at 2026-09-25 04:59:35 on dist-test-slave-xzjr"
server_key: "ca917d0e57cd5d85aa91ab5ac7fb6b10"
server_key_iv: "42505e15edd47800d2343bb72be1037d"
server_key_version: "encryptionkey@0"
I20260925 04:59:35.800612 25691 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260925 04:59:35.804042 25706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:35.805449 25691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 04:59:35.805624 25691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-0/wal
uuid: "e6567b3b47a2425483a479140fdad1ef"
format_stamp: "Formatted at 2026-09-25 04:59:35 on dist-test-slave-xzjr"
server_key: "ca917d0e57cd5d85aa91ab5ac7fb6b10"
server_key_iv: "42505e15edd47800d2343bb72be1037d"
server_key_version: "encryptionkey@0"
I20260925 04:59:35.805831 25691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:59:35.828624 25691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:59:35.833364 25691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:59:35.833688 25691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:59:35.834746 25691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:59:35.836164 25691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:59:35.836275 25691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:35.836431 25691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:59:35.836505 25691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:35.871635 25691 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.129:41765
I20260925 04:59:35.871690 25818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.129:41765 every 8 connection(s)
I20260925 04:59:35.872995 25691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
I20260925 04:59:35.877714 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 25691
I20260925 04:59:35.877897 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance
I20260925 04:59:35.878367 6262 external_mini_cluster.cc:1428] Setting key e0bb57247de777af80bb8170edd1413a
Sep 25 04:59:35 dist-test-slave-xzjr krb5kdc[25585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312375, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.129@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:59:35.893105 25821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:35.875533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:36817 (local address 127.6.29.190:43545)
0925 04:59:35.876034 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:35.876039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:35.876867 (+ 828us) server_negotiation.cc:407] Connection header received
0925 04:59:35.878970 (+ 2103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:35.878976 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:35.879063 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:35.879197 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:35.880789 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:35.881509 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:35.883119 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:35.883531 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:35.883683 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:35.887242 (+ 3559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:35.887251 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:35.887260 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:59:35.887295 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:35.889895 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:35.890647 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:35.890657 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:35.890659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:35.890782 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:35.891245 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:35.891249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:35.891252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:35.891518 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:35.891697 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:35.892491 (+ 794us) server_negotiation.cc:299] Negotiation successful
0925 04:59:35.892738 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":276,"threads_started":1}
I20260925 04:59:35.894140 25822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:35.876006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43545 (local address 127.6.29.129:36817)
0925 04:59:35.876698 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:35.876736 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:35.878688 (+ 1952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:35.879449 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:35.879458 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:35.879993 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:59:35.880607 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:35.880625 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:35.881702 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:35.881707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:35.882961 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:35.882973 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:35.883817 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:35.883827 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:35.884070 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:35.884795 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:59:35.884817 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:59:35.887011 (+ 2194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:59:35.890043 (+ 3032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:59:35.890053 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:59:35.890069 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:59:35.890508 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:59:35.890899 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:59:35.890904 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:59:35.890908 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:59:35.891111 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:59:35.891667 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:59:35.891675 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:59:35.892104 (+ 429us) client_negotiation.cc:769] Sending connection context
0925 04:59:35.892376 (+ 272us) client_negotiation.cc:240] Negotiation successful
0925 04:59:35.892932 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":258,"threads_started":1}
I20260925 04:59:35.895844 25819 heartbeater.cc:347] Connected to a master server at 127.6.29.190:43545
I20260925 04:59:35.896185 25819 heartbeater.cc:464] Registering TS with master...
I20260925 04:59:35.897084 25819 heartbeater.cc:511] Master 127.6.29.190:43545 requested a full tablet report, sending...
I20260925 04:59:35.899856 25630 ts_manager.cc:194] Registered new tserver with Master: e6567b3b47a2425483a479140fdad1ef (127.6.29.129:41765)
I20260925 04:59:35.902064 25630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.129@KRBTEST.COM'} at 127.6.29.129:36817
WARNING: no policy specified for kudu/127.6.29.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab.
Entry for principal kudu/127.6.29.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab.
I20260925 04:59:35.936518 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.130:0
--local_ip_for_outbound_sockets=127.6.29.130
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:43545
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:33537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:59:36.274490 25825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:59:36.274758 25825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:59:36.274938 25825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:59:36.282143 25825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:59:36.282253 25825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:59:36.282438 25825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.130
I20260925 04:59:36.292133 25825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:33537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.130.keytab
--principal=kudu/127.6.29.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:43545
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.130
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:59:36.293447 25825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:59:36.294803 25825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:59:36.308759 25830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:36.308786 25831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:36.308861 25833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:59:36.309696 25825 server_base.cc:1034] running on GCE node
I20260925 04:59:36.310326 25825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:59:36.311324 25825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:59:36.312577 25825 hybrid_clock.cc:648] HybridClock initialized: now 1790312376312508 us; error 58 us; skew 500 ppm
I20260925 04:59:36.312906 25825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:59:36 dist-test-slave-xzjr krb5kdc[25585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312376, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:59:36.316093 25825 init.cc:377] Logged in from keytab as kudu/127.6.29.130@KRBTEST.COM (short username kudu)
I20260925 04:59:36.317499 25825 webserver.cc:460] Webserver started at http://127.6.29.130:40543/ using document root <none> and password file <none>
I20260925 04:59:36.318208 25825 fs_manager.cc:359] Metadata directory not provided
I20260925 04:59:36.318360 25825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:59:36.318684 25825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:59:36.321682 25825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-1/data/instance:
uuid: "03d6b5ceed4f4b2bad9a0b07ccc9df44"
format_stamp: "Formatted at 2026-09-25 04:59:36 on dist-test-slave-xzjr"
server_key: "7ab5d72b7449d371e2f3daa76fb0c325"
server_key_iv: "ebc7cbe2393276ffb43d3351eb46b226"
server_key_version: "encryptionkey@0"
I20260925 04:59:36.322475 25825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance:
uuid: "03d6b5ceed4f4b2bad9a0b07ccc9df44"
format_stamp: "Formatted at 2026-09-25 04:59:36 on dist-test-slave-xzjr"
server_key: "7ab5d72b7449d371e2f3daa76fb0c325"
server_key_iv: "ebc7cbe2393276ffb43d3351eb46b226"
server_key_version: "encryptionkey@0"
I20260925 04:59:36.327641 25825 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 04:59:36.331041 25840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:36.332414 25825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:59:36.332590 25825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-1/wal
uuid: "03d6b5ceed4f4b2bad9a0b07ccc9df44"
format_stamp: "Formatted at 2026-09-25 04:59:36 on dist-test-slave-xzjr"
server_key: "7ab5d72b7449d371e2f3daa76fb0c325"
server_key_iv: "ebc7cbe2393276ffb43d3351eb46b226"
server_key_version: "encryptionkey@0"
I20260925 04:59:36.332791 25825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:59:36.371364 25825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:59:36.375928 25825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:59:36.376261 25825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:59:36.377295 25825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:59:36.378717 25825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:59:36.378821 25825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:36.378974 25825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:59:36.379046 25825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:36.412009 25825 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.130:36287
I20260925 04:59:36.412070 25952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.130:36287 every 8 connection(s)
I20260925 04:59:36.413314 25825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
I20260925 04:59:36.422029 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 25825
I20260925 04:59:36.422298 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance
I20260925 04:59:36.422842 6262 external_mini_cluster.cc:1428] Setting key 509ffd015e63f95bc8d9f08d459ae90f
Sep 25 04:59:36 dist-test-slave-xzjr krb5kdc[25585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312376, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.130@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:59:36.436681 25955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:36.416030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:40639 (local address 127.6.29.190:43545)
0925 04:59:36.416464 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:36.416468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:36.417488 (+ 1020us) server_negotiation.cc:407] Connection header received
0925 04:59:36.419286 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:36.419290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:36.419372 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:36.419475 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:36.421132 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:36.421889 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:36.423989 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:36.424682 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:36.424842 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:36.429030 (+ 4188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:36.429039 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:36.429047 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:59:36.429092 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:36.433097 (+ 4005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:36.433925 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:36.433933 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:36.433939 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:36.434116 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:36.434668 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:36.434672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:36.434675 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:36.434974 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:36.435223 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:36.436020 (+ 797us) server_negotiation.cc:299] Negotiation successful
0925 04:59:36.436275 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":214,"threads_started":1}
I20260925 04:59:36.438009 25956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:36.416609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43545 (local address 127.6.29.130:40639)
0925 04:59:36.417314 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:36.417361 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:36.419002 (+ 1641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:36.419703 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:36.419711 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:36.420366 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:59:36.420935 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:36.420953 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:36.422036 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:36.422041 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:36.423861 (+ 1820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:36.423873 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:36.424901 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:36.424906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:36.425198 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:36.425930 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:59:36.425966 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:59:36.428778 (+ 2812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:59:36.433270 (+ 4492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:59:36.433278 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:59:36.433293 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:59:36.433778 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:59:36.434256 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:59:36.434260 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:59:36.434263 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:59:36.434477 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:59:36.435134 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:59:36.435143 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:59:36.435544 (+ 401us) client_negotiation.cc:769] Sending connection context
0925 04:59:36.435873 (+ 329us) client_negotiation.cc:240] Negotiation successful
0925 04:59:36.436485 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":286,"threads_started":1}
I20260925 04:59:36.439669 25953 heartbeater.cc:347] Connected to a master server at 127.6.29.190:43545
I20260925 04:59:36.440038 25953 heartbeater.cc:464] Registering TS with master...
I20260925 04:59:36.440830 25953 heartbeater.cc:511] Master 127.6.29.190:43545 requested a full tablet report, sending...
I20260925 04:59:36.442430 25630 ts_manager.cc:194] Registered new tserver with Master: 03d6b5ceed4f4b2bad9a0b07ccc9df44 (127.6.29.130:36287)
I20260925 04:59:36.443701 25630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.130@KRBTEST.COM'} at 127.6.29.130:40639
WARNING: no policy specified for kudu/127.6.29.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab.
Entry for principal kudu/127.6.29.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab.
I20260925 04:59:36.482957 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.131:0
--local_ip_for_outbound_sockets=127.6.29.131
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:43545
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:33537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:59:36.814381 25959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:59:36.814713 25959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:59:36.814869 25959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:59:36.822616 25959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:59:36.822773 25959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:59:36.822952 25959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.131
I20260925 04:59:36.832726 25959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:33537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.131.keytab
--principal=kudu/127.6.29.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:43545
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.131
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:59:36.834018 25959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:59:36.835332 25959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:59:36.848733 25965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:36.848778 25967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:36.848755 25964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:59:36.850081 25959 server_base.cc:1034] running on GCE node
I20260925 04:59:36.850777 25959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:59:36.851816 25959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:59:36.853039 25959 hybrid_clock.cc:648] HybridClock initialized: now 1790312376852984 us; error 43 us; skew 500 ppm
I20260925 04:59:36.853446 25959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:59:36 dist-test-slave-xzjr krb5kdc[25585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312376, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:59:36.856778 25959 init.cc:377] Logged in from keytab as kudu/127.6.29.131@KRBTEST.COM (short username kudu)
I20260925 04:59:36.858337 25959 webserver.cc:460] Webserver started at http://127.6.29.131:34061/ using document root <none> and password file <none>
I20260925 04:59:36.859016 25959 fs_manager.cc:359] Metadata directory not provided
I20260925 04:59:36.859169 25959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:59:36.859503 25959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:59:36.862494 25959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-2/data/instance:
uuid: "0262c1074a234ebf97c36f424ab554a2"
format_stamp: "Formatted at 2026-09-25 04:59:36 on dist-test-slave-xzjr"
server_key: "31ebb90880ee827e2f18c702ba4345cc"
server_key_iv: "a77c344d01fd720df9b5a0a1953d7bee"
server_key_version: "encryptionkey@0"
I20260925 04:59:36.863313 25959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance:
uuid: "0262c1074a234ebf97c36f424ab554a2"
format_stamp: "Formatted at 2026-09-25 04:59:36 on dist-test-slave-xzjr"
server_key: "31ebb90880ee827e2f18c702ba4345cc"
server_key_iv: "a77c344d01fd720df9b5a0a1953d7bee"
server_key_version: "encryptionkey@0"
I20260925 04:59:36.868858 25959 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260925 04:59:36.873092 25974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:36.874545 25959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260925 04:59:36.874709 25959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-2/wal
uuid: "0262c1074a234ebf97c36f424ab554a2"
format_stamp: "Formatted at 2026-09-25 04:59:36 on dist-test-slave-xzjr"
server_key: "31ebb90880ee827e2f18c702ba4345cc"
server_key_iv: "a77c344d01fd720df9b5a0a1953d7bee"
server_key_version: "encryptionkey@0"
I20260925 04:59:36.874902 25959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:59:36.895391 25959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:59:36.899879 25959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:59:36.900214 25959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:59:36.901202 25959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:59:36.902575 25959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:59:36.902650 25959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:36.902773 25959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:59:36.902833 25959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:36.905447 25819 heartbeater.cc:503] Master 127.6.29.190:43545 was elected leader, sending a full tablet report...
I20260925 04:59:36.937616 25959 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.131:43973
I20260925 04:59:36.937649 26086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.131:43973 every 8 connection(s)
I20260925 04:59:36.939070 25959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
I20260925 04:59:36.947342 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 25959
I20260925 04:59:36.947520 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance
I20260925 04:59:36.948117 6262 external_mini_cluster.cc:1428] Setting key 1bc19322aac4a8540532ed2890696fe6
Sep 25 04:59:36 dist-test-slave-xzjr krb5kdc[25585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312376, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.131@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 04:59:36.959317 26089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:36.941727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:43391 (local address 127.6.29.190:43545)
0925 04:59:36.942236 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:36.942241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:36.943155 (+ 914us) server_negotiation.cc:407] Connection header received
0925 04:59:36.945037 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:36.945042 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:36.945123 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:36.945218 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:36.946636 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:36.947745 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:36.949630 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:36.950245 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:36.950368 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:36.953835 (+ 3467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:36.953840 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:36.953843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:59:36.953876 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:36.956554 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:36.957193 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:36.957196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:36.957199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:36.957314 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:36.957738 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:36.957742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:36.957744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:36.957994 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:36.958125 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:36.958859 (+ 734us) server_negotiation.cc:299] Negotiation successful
0925 04:59:36.959043 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":235,"threads_started":1}
I20260925 04:59:36.960356 26090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:36.942308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43545 (local address 127.6.29.131:43391)
0925 04:59:36.942984 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:36.943029 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:36.944773 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:36.945430 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:36.945438 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:36.945924 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:59:36.946494 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:36.946510 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:36.947893 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:36.947896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:36.949497 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:36.949508 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:36.950454 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:36.950459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:36.950649 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:36.951395 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 04:59:36.951422 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:59:36.953625 (+ 2203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:59:36.956675 (+ 3050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:59:36.956682 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:59:36.956694 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:59:36.957088 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:59:36.957438 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 04:59:36.957442 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 04:59:36.957445 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 04:59:36.957634 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 04:59:36.958178 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:59:36.958185 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:59:36.958504 (+ 319us) client_negotiation.cc:769] Sending connection context
0925 04:59:36.958745 (+ 241us) client_negotiation.cc:240] Negotiation successful
0925 04:59:36.959162 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":256,"threads_started":1}
I20260925 04:59:36.961882 26087 heartbeater.cc:347] Connected to a master server at 127.6.29.190:43545
I20260925 04:59:36.962255 26087 heartbeater.cc:464] Registering TS with master...
I20260925 04:59:36.963045 26087 heartbeater.cc:511] Master 127.6.29.190:43545 requested a full tablet report, sending...
I20260925 04:59:36.964599 25630 ts_manager.cc:194] Registered new tserver with Master: 0262c1074a234ebf97c36f424ab554a2 (127.6.29.131:43973)
I20260925 04:59:36.965473 25630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.29.131@KRBTEST.COM'} at 127.6.29.131:43391
I20260925 04:59:36.975850 6262 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:59:36.992261 26089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:36.980074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59370 (local address 127.6.29.190:43545)
0925 04:59:36.980271 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:36.980275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:36.980782 (+ 507us) server_negotiation.cc:407] Connection header received
0925 04:59:36.980912 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:36.980916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:36.980985 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:36.981066 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:36.982397 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:36.983141 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:36.984470 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:36.984856 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:36.984949 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:36.987230 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:36.987235 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:36.987238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:59:36.987271 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:36.989848 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:36.990437 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:36.990441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:36.990443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:36.990574 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:36.990904 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:36.990907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:36.990909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:36.991269 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:36.991476 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:36.991824 (+ 348us) server_negotiation.cc:299] Negotiation successful
0925 04:59:36.991998 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260925 04:59:36.996021 25630 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:59:37.002766 25630 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:59:37.038149 26102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:37.014960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:41765 (local address 127.0.0.1:60826)
0925 04:59:37.029456 (+ 14496us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:37.029487 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:37.029613 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:37.030384 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:37.030393 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:37.030424 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:59:37.030815 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:37.030828 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:37.032658 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:37.032664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:37.035437 (+ 2773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:37.035451 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:37.037191 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:37.037198 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:37.037384 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:37.037435 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 04:59:37.037545 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 04:59:37.037738 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14335,"mutex_wait_us":46,"thread_start_us":14976,"threads_started":1}
I20260925 04:59:37.038277 26105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:37.017563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54936 (local address 127.6.29.131:43973)
0925 04:59:37.027687 (+ 10124us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:37.027700 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:37.030152 (+ 2452us) server_negotiation.cc:407] Connection header received
0925 04:59:37.030433 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:37.030440 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:37.030695 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:37.030853 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:59:37.031822 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:37.033403 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:37.035395 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:37.036782 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:37.036907 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:37.037162 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:37.037723 (+ 561us) server_negotiation.cc:299] Negotiation successful
0925 04:59:37.037903 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9885,"thread_start_us":9707,"threads_started":1}
I20260925 04:59:37.038820 26104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:37.014771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:36287 (local address 127.0.0.1:51780)
0925 04:59:37.028426 (+ 13655us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:37.028491 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:37.028747 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:37.030833 (+ 2086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:37.030837 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:37.030865 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:59:37.031349 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:37.031364 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:37.033341 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:37.033350 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:37.035755 (+ 2405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:37.035791 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:37.037926 (+ 2135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:37.037932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:37.038158 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:37.038189 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 04:59:37.038280 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 04:59:37.038413 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13305,"mutex_wait_us":24,"thread_start_us":15794,"threads_started":1}
I20260925 04:59:37.039326 26100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:37.016906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.131:43973 (local address 127.0.0.1:54936)
0925 04:59:37.030029 (+ 13123us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:37.030051 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:37.030257 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:37.031149 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:37.031154 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:37.031178 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:59:37.031598 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:37.031609 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:37.033571 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:37.033579 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:37.035230 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:37.035245 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:37.037320 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:37.037327 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:37.037501 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:37.037543 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 04:59:37.038533 (+ 990us) client_negotiation.cc:240] Negotiation successful
0925 04:59:37.038667 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13018,"mutex_wait_us":22,"thread_start_us":11003,"threads_started":1}
I20260925 04:59:37.039341 26101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:37.015036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51780 (local address 127.6.29.130:36287)
0925 04:59:37.030250 (+ 15214us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:37.030258 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:37.030310 (+ 52us) server_negotiation.cc:407] Connection header received
0925 04:59:37.030412 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:37.030419 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:37.030629 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:37.030851 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:59:37.031540 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:37.033177 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:37.035926 (+ 2749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:37.037438 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:37.037564 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:37.037776 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:37.038924 (+ 1148us) server_negotiation.cc:299] Negotiation successful
0925 04:59:37.039114 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15059,"thread_start_us":16053,"threads_started":1}
I20260925 04:59:37.039431 26103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:37.015342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60826 (local address 127.6.29.129:41765)
0925 04:59:37.027056 (+ 11714us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:37.027065 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:37.029592 (+ 2527us) server_negotiation.cc:407] Connection header received
0925 04:59:37.029778 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:37.029790 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:37.030135 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:37.030350 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:59:37.031028 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:37.032494 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:37.035614 (+ 3120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:37.036964 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:37.038681 (+ 1717us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:37.038836 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:37.038931 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 04:59:37.039084 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11521,"thread_start_us":11364,"threads_started":1}
I20260925 04:59:37.043318 25888 tablet_service.cc:1467] Processing CreateTablet for tablet bb3f4334a0554a2491987ef93a3ddb35 (DEFAULT_TABLE table=test-table [id=dd73091202714cc09478bf5599c2d56e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:59:37.043318 25754 tablet_service.cc:1467] Processing CreateTablet for tablet bb3f4334a0554a2491987ef93a3ddb35 (DEFAULT_TABLE table=test-table [id=dd73091202714cc09478bf5599c2d56e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:59:37.044885 25888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb3f4334a0554a2491987ef93a3ddb35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:59:37.044888 25754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb3f4334a0554a2491987ef93a3ddb35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:59:37.050345 26022 tablet_service.cc:1467] Processing CreateTablet for tablet bb3f4334a0554a2491987ef93a3ddb35 (DEFAULT_TABLE table=test-table [id=dd73091202714cc09478bf5599c2d56e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:59:37.051523 26022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb3f4334a0554a2491987ef93a3ddb35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:59:37.059995 26106 tablet_bootstrap.cc:492] T bb3f4334a0554a2491987ef93a3ddb35 P e6567b3b47a2425483a479140fdad1ef: Bootstrap starting.
I20260925 04:59:37.060770 26107 tablet_bootstrap.cc:492] T bb3f4334a0554a2491987ef93a3ddb35 P 03d6b5ceed4f4b2bad9a0b07ccc9df44: Bootstrap starting.
I20260925 04:59:37.063910 26108 tablet_bootstrap.cc:492] T bb3f4334a0554a2491987ef93a3ddb35 P 0262c1074a234ebf97c36f424ab554a2: Bootstrap starting.
I20260925 04:59:37.064280 26106 tablet_bootstrap.cc:654] T bb3f4334a0554a2491987ef93a3ddb35 P e6567b3b47a2425483a479140fdad1ef: Neither blocks nor log segments found. Creating new log.
I20260925 04:59:37.065330 26107 tablet_bootstrap.cc:654] T bb3f4334a0554a2491987ef93a3ddb35 P 03d6b5ceed4f4b2bad9a0b07ccc9df44: Neither blocks nor log segments found. Creating new log.
I20260925 04:59:37.065666 26106 log.cc:824] T bb3f4334a0554a2491987ef93a3ddb35 P e6567b3b47a2425483a479140fdad1ef: Log is configured to *not* fsync() on all Append() calls
I20260925 04:59:37.066537 26107 log.cc:824] T bb3f4334a0554a2491987ef93a3ddb35 P 03d6b5ceed4f4b2bad9a0b07ccc9df44: Log is configured to *not* fsync() on all Append() calls
I20260925 04:59:37.067418 26108 tablet_bootstrap.cc:654] T bb3f4334a0554a2491987ef93a3ddb35 P 0262c1074a234ebf97c36f424ab554a2: Neither blocks nor log segments found. Creating new log.
I20260925 04:59:37.068652 26108 log.cc:824] T bb3f4334a0554a2491987ef93a3ddb35 P 0262c1074a234ebf97c36f424ab554a2: Log is configured to *not* fsync() on all Append() calls
I20260925 04:59:37.068992 26106 tablet_bootstrap.cc:492] T bb3f4334a0554a2491987ef93a3ddb35 P e6567b3b47a2425483a479140fdad1ef: No bootstrap required, opened a new log
I20260925 04:59:37.069331 26106 ts_tablet_manager.cc:1397] T bb3f4334a0554a2491987ef93a3ddb35 P e6567b3b47a2425483a479140fdad1ef: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.001s
I20260925 04:59:37.074425 26107 tablet_bootstrap.cc:492] T bb3f4334a0554a2491987ef93a3ddb35 P 03d6b5ceed4f4b2bad9a0b07ccc9df44: No bootstrap required, opened a new log
I20260925 04:59:37.074455 26108 tablet_bootstrap.cc:492] T bb3f4334a0554a2491987ef93a3ddb35 P 0262c1074a234ebf97c36f424ab554a2: No bootstrap required, opened a new log
I20260925 04:59:37.074738 26108 ts_tablet_manager.cc:1397] T bb3f4334a0554a2491987ef93a3ddb35 P 0262c1074a234ebf97c36f424ab554a2: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260925 04:59:37.074769 26107 ts_tablet_manager.cc:1397] T bb3f4334a0554a2491987ef93a3ddb35 P 03d6b5ceed4f4b2bad9a0b07ccc9df44: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260925 04:59:37.078621 26106 raft_consensus.cc:348] T bb3f4334a0554a2491987ef93a3ddb35 P e6567b3b47a2425483a479140fdad1ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d6b5ceed4f4b2bad9a0b07ccc9df44" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 36287 } } peers { permanent_uuid: "e6567b3b47a2425483a479140fdad1ef" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41765 } } peers { permanent_uuid: "0262c1074a234ebf97c36f424ab554a2" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43973 } }
I20260925 04:59:37.079084 26106 raft_consensus.cc:374] T bb3f4334a0554a2491987ef93a3ddb35 P e6567b3b47a2425483a479140fdad1ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:59:37.079195 26106 raft_consensus.cc:729] T bb3f4334a0554a2491987ef93a3ddb35 P e6567b3b47a2425483a479140fdad1ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6567b3b47a2425483a479140fdad1ef, State: Initialized, Role: FOLLOWER
I20260925 04:59:37.079926 26106 consensus_queue.cc:260] T bb3f4334a0554a2491987ef93a3ddb35 P e6567b3b47a2425483a479140fdad1ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d6b5ceed4f4b2bad9a0b07ccc9df44" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 36287 } } peers { permanent_uuid: "e6567b3b47a2425483a479140fdad1ef" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41765 } } peers { permanent_uuid: "0262c1074a234ebf97c36f424ab554a2" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43973 } }
I20260925 04:59:37.081744 26106 ts_tablet_manager.cc:1428] T bb3f4334a0554a2491987ef93a3ddb35 P e6567b3b47a2425483a479140fdad1ef: Time spent starting tablet: real 0.012s user 0.010s sys 0.002s
I20260925 04:59:37.085675 26108 raft_consensus.cc:348] T bb3f4334a0554a2491987ef93a3ddb35 P 0262c1074a234ebf97c36f424ab554a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d6b5ceed4f4b2bad9a0b07ccc9df44" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 36287 } } peers { permanent_uuid: "e6567b3b47a2425483a479140fdad1ef" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41765 } } peers { permanent_uuid: "0262c1074a234ebf97c36f424ab554a2" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43973 } }
I20260925 04:59:37.085665 26107 raft_consensus.cc:348] T bb3f4334a0554a2491987ef93a3ddb35 P 03d6b5ceed4f4b2bad9a0b07ccc9df44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d6b5ceed4f4b2bad9a0b07ccc9df44" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 36287 } } peers { permanent_uuid: "e6567b3b47a2425483a479140fdad1ef" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41765 } } peers { permanent_uuid: "0262c1074a234ebf97c36f424ab554a2" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43973 } }
I20260925 04:59:37.086122 26108 raft_consensus.cc:374] T bb3f4334a0554a2491987ef93a3ddb35 P 0262c1074a234ebf97c36f424ab554a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:59:37.086122 26107 raft_consensus.cc:374] T bb3f4334a0554a2491987ef93a3ddb35 P 03d6b5ceed4f4b2bad9a0b07ccc9df44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:59:37.086233 26108 raft_consensus.cc:729] T bb3f4334a0554a2491987ef93a3ddb35 P 0262c1074a234ebf97c36f424ab554a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0262c1074a234ebf97c36f424ab554a2, State: Initialized, Role: FOLLOWER
I20260925 04:59:37.086233 26107 raft_consensus.cc:729] T bb3f4334a0554a2491987ef93a3ddb35 P 03d6b5ceed4f4b2bad9a0b07ccc9df44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03d6b5ceed4f4b2bad9a0b07ccc9df44, State: Initialized, Role: FOLLOWER
I20260925 04:59:37.086861 26108 consensus_queue.cc:260] T bb3f4334a0554a2491987ef93a3ddb35 P 0262c1074a234ebf97c36f424ab554a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d6b5ceed4f4b2bad9a0b07ccc9df44" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 36287 } } peers { permanent_uuid: "e6567b3b47a2425483a479140fdad1ef" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41765 } } peers { permanent_uuid: "0262c1074a234ebf97c36f424ab554a2" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43973 } }
I20260925 04:59:37.086855 26107 consensus_queue.cc:260] T bb3f4334a0554a2491987ef93a3ddb35 P 03d6b5ceed4f4b2bad9a0b07ccc9df44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d6b5ceed4f4b2bad9a0b07ccc9df44" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 36287 } } peers { permanent_uuid: "e6567b3b47a2425483a479140fdad1ef" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41765 } } peers { permanent_uuid: "0262c1074a234ebf97c36f424ab554a2" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43973 } }
I20260925 04:59:37.088651 25953 heartbeater.cc:503] Master 127.6.29.190:43545 was elected leader, sending a full tablet report...
I20260925 04:59:37.088810 26108 ts_tablet_manager.cc:1428] T bb3f4334a0554a2491987ef93a3ddb35 P 0262c1074a234ebf97c36f424ab554a2: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260925 04:59:37.089252 26107 ts_tablet_manager.cc:1428] T bb3f4334a0554a2491987ef93a3ddb35 P 03d6b5ceed4f4b2bad9a0b07ccc9df44: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260925 04:59:37.089464 26087 heartbeater.cc:503] Master 127.6.29.190:43545 was elected leader, sending a full tablet report...
W20260925 04:59:37.125494 25820 tablet.cc:2369] T bb3f4334a0554a2491987ef93a3ddb35 P e6567b3b47a2425483a479140fdad1ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:59:37.165684 25954 tablet.cc:2369] T bb3f4334a0554a2491987ef93a3ddb35 P 03d6b5ceed4f4b2bad9a0b07ccc9df44: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 04:59:37.190780 26088 tablet.cc:2369] T bb3f4334a0554a2491987ef93a3ddb35 P 0262c1074a234ebf97c36f424ab554a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 04:59:37.232769 26112 raft_consensus.cc:482] T bb3f4334a0554a2491987ef93a3ddb35 P e6567b3b47a2425483a479140fdad1ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:59:37.233107 26112 raft_consensus.cc:504] T bb3f4334a0554a2491987ef93a3ddb35 P e6567b3b47a2425483a479140fdad1ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d6b5ceed4f4b2bad9a0b07ccc9df44" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 36287 } } peers { permanent_uuid: "e6567b3b47a2425483a479140fdad1ef" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41765 } } peers { permanent_uuid: "0262c1074a234ebf97c36f424ab554a2" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43973 } }
I20260925 04:59:37.234889 26112 leader_election.cc:288] T bb3f4334a0554a2491987ef93a3ddb35 P e6567b3b47a2425483a479140fdad1ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03d6b5ceed4f4b2bad9a0b07ccc9df44 (127.6.29.130:36287), 0262c1074a234ebf97c36f424ab554a2 (127.6.29.131:43973)
I20260925 04:59:37.241978 26105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:37.235458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:40503 (local address 127.6.29.131:43973)
0925 04:59:37.235687 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:37.235696 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:37.236351 (+ 655us) server_negotiation.cc:407] Connection header received
0925 04:59:37.236618 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:37.236622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:37.236684 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:37.236787 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:59:37.237523 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:37.238263 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:37.240229 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:37.240976 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:37.241148 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:37.241231 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:37.241575 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 04:59:37.241709 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 04:59:37.241990 26101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:37.235269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:53141 (local address 127.6.29.130:36287)
0925 04:59:37.235542 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:37.235549 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:37.236468 (+ 919us) server_negotiation.cc:407] Connection header received
0925 04:59:37.236676 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:37.236680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:37.236739 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:37.236917 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:59:37.237707 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:37.238396 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:37.240244 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:37.241024 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:37.241199 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:37.241280 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:37.241579 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 04:59:37.241724 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260925 04:59:37.242162 26115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:37.235181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.131:43973 (local address 127.6.29.129:40503)
0925 04:59:37.236223 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:37.236243 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:37.236466 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:37.236819 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:37.236824 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:37.236859 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:59:37.237367 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:37.237378 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:37.238393 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:37.238401 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:37.240069 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:37.240081 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:37.241212 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:37.241218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:37.241374 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:37.241441 (+ 67us) client_negotiation.cc:769] Sending connection context
0925 04:59:37.241538 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 04:59:37.241699 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":918,"mutex_wait_us":109,"thread_start_us":637,"threads_started":1}
I20260925 04:59:37.242213 26116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:37.235195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:36287 (local address 127.6.29.129:53141)
0925 04:59:37.236359 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:37.236375 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:37.236522 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:37.236935 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:37.236940 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:37.236982 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:59:37.237533 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:37.237560 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:37.238540 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:37.238545 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:37.240094 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:37.240112 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:37.241254 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:37.241260 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:37.241430 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:37.241452 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:59:37.241554 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 04:59:37.241743 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1065,"mutex_wait_us":66,"spinlock_wait_cycles":896,"thread_start_us":696,"threads_started":1}
I20260925 04:59:37.243402 25908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb3f4334a0554a2491987ef93a3ddb35" candidate_uuid: "e6567b3b47a2425483a479140fdad1ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03d6b5ceed4f4b2bad9a0b07ccc9df44" is_pre_election: true
I20260925 04:59:37.243368 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb3f4334a0554a2491987ef93a3ddb35" candidate_uuid: "e6567b3b47a2425483a479140fdad1ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0262c1074a234ebf97c36f424ab554a2" is_pre_election: true
I20260925 04:59:37.244009 26042 raft_consensus.cc:2454] T bb3f4334a0554a2491987ef93a3ddb35 P 0262c1074a234ebf97c36f424ab554a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6567b3b47a2425483a479140fdad1ef in term 0.
I20260925 04:59:37.244009 25908 raft_consensus.cc:2454] T bb3f4334a0554a2491987ef93a3ddb35 P 03d6b5ceed4f4b2bad9a0b07ccc9df44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6567b3b47a2425483a479140fdad1ef in term 0.
I20260925 04:59:37.244819 25707 leader_election.cc:302] T bb3f4334a0554a2491987ef93a3ddb35 P e6567b3b47a2425483a479140fdad1ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0262c1074a234ebf97c36f424ab554a2, 03d6b5ceed4f4b2bad9a0b07ccc9df44, e6567b3b47a2425483a479140fdad1ef; no voters:
I20260925 04:59:37.245312 26112 raft_consensus.cc:2781] T bb3f4334a0554a2491987ef93a3ddb35 P e6567b3b47a2425483a479140fdad1ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:59:37.245451 26112 raft_consensus.cc:482] T bb3f4334a0554a2491987ef93a3ddb35 P e6567b3b47a2425483a479140fdad1ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:59:37.245564 26112 raft_consensus.cc:3037] T bb3f4334a0554a2491987ef93a3ddb35 P e6567b3b47a2425483a479140fdad1ef [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:59:37.247923 26112 raft_consensus.cc:504] T bb3f4334a0554a2491987ef93a3ddb35 P e6567b3b47a2425483a479140fdad1ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d6b5ceed4f4b2bad9a0b07ccc9df44" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 36287 } } peers { permanent_uuid: "e6567b3b47a2425483a479140fdad1ef" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41765 } } peers { permanent_uuid: "0262c1074a234ebf97c36f424ab554a2" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43973 } }
I20260925 04:59:37.248575 26112 leader_election.cc:288] T bb3f4334a0554a2491987ef93a3ddb35 P e6567b3b47a2425483a479140fdad1ef [CANDIDATE]: Term 1 election: Requested vote from peers 03d6b5ceed4f4b2bad9a0b07ccc9df44 (127.6.29.130:36287), 0262c1074a234ebf97c36f424ab554a2 (127.6.29.131:43973)
I20260925 04:59:37.249063 26042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb3f4334a0554a2491987ef93a3ddb35" candidate_uuid: "e6567b3b47a2425483a479140fdad1ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0262c1074a234ebf97c36f424ab554a2"
I20260925 04:59:37.249099 25908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb3f4334a0554a2491987ef93a3ddb35" candidate_uuid: "e6567b3b47a2425483a479140fdad1ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03d6b5ceed4f4b2bad9a0b07ccc9df44"
I20260925 04:59:37.249269 26042 raft_consensus.cc:3037] T bb3f4334a0554a2491987ef93a3ddb35 P 0262c1074a234ebf97c36f424ab554a2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:59:37.249315 25908 raft_consensus.cc:3037] T bb3f4334a0554a2491987ef93a3ddb35 P 03d6b5ceed4f4b2bad9a0b07ccc9df44 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:59:37.251607 25908 raft_consensus.cc:2454] T bb3f4334a0554a2491987ef93a3ddb35 P 03d6b5ceed4f4b2bad9a0b07ccc9df44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6567b3b47a2425483a479140fdad1ef in term 1.
I20260925 04:59:37.251658 26042 raft_consensus.cc:2454] T bb3f4334a0554a2491987ef93a3ddb35 P 0262c1074a234ebf97c36f424ab554a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6567b3b47a2425483a479140fdad1ef in term 1.
I20260925 04:59:37.252102 25710 leader_election.cc:302] T bb3f4334a0554a2491987ef93a3ddb35 P e6567b3b47a2425483a479140fdad1ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03d6b5ceed4f4b2bad9a0b07ccc9df44, e6567b3b47a2425483a479140fdad1ef; no voters:
I20260925 04:59:37.252444 26112 raft_consensus.cc:2781] T bb3f4334a0554a2491987ef93a3ddb35 P e6567b3b47a2425483a479140fdad1ef [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:59:37.252931 26112 raft_consensus.cc:686] T bb3f4334a0554a2491987ef93a3ddb35 P e6567b3b47a2425483a479140fdad1ef [term 1 LEADER]: Becoming Leader. State: Replica: e6567b3b47a2425483a479140fdad1ef, State: Running, Role: LEADER
I20260925 04:59:37.253455 26112 consensus_queue.cc:237] T bb3f4334a0554a2491987ef93a3ddb35 P e6567b3b47a2425483a479140fdad1ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d6b5ceed4f4b2bad9a0b07ccc9df44" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 36287 } } peers { permanent_uuid: "e6567b3b47a2425483a479140fdad1ef" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41765 } } peers { permanent_uuid: "0262c1074a234ebf97c36f424ab554a2" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43973 } }
I20260925 04:59:37.258306 25630 catalog_manager.cc:5495] T bb3f4334a0554a2491987ef93a3ddb35 P e6567b3b47a2425483a479140fdad1ef reported cstate change: term changed from 0 to 1, leader changed from <none> to e6567b3b47a2425483a479140fdad1ef (127.6.29.129). New cstate: current_term: 1 leader_uuid: "e6567b3b47a2425483a479140fdad1ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03d6b5ceed4f4b2bad9a0b07ccc9df44" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 36287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6567b3b47a2425483a479140fdad1ef" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 41765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0262c1074a234ebf97c36f424ab554a2" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43973 } health_report { overall_health: UNKNOWN } } }
I20260925 04:59:37.274653 26103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:37.268926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60840 (local address 127.6.29.129:41765)
0925 04:59:37.269182 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:37.269188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:37.269291 (+ 103us) server_negotiation.cc:407] Connection header received
0925 04:59:37.269538 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:37.269542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:37.269621 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:37.269793 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:59:37.270159 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:37.270884 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:37.272438 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:37.272879 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:37.273009 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:37.273546 (+ 537us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:59:37.274087 (+ 541us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:59:37.274260 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:37.274309 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 04:59:37.274391 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 04:59:37.283149 25908 raft_consensus.cc:1261] T bb3f4334a0554a2491987ef93a3ddb35 P 03d6b5ceed4f4b2bad9a0b07ccc9df44 [term 1 FOLLOWER]: Refusing update from remote peer e6567b3b47a2425483a479140fdad1ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:59:37.283159 26042 raft_consensus.cc:1261] T bb3f4334a0554a2491987ef93a3ddb35 P 0262c1074a234ebf97c36f424ab554a2 [term 1 FOLLOWER]: Refusing update from remote peer e6567b3b47a2425483a479140fdad1ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:59:37.284137 26117 consensus_queue.cc:1035] T bb3f4334a0554a2491987ef93a3ddb35 P e6567b3b47a2425483a479140fdad1ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "03d6b5ceed4f4b2bad9a0b07ccc9df44" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 36287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:59:37.284499 26112 consensus_queue.cc:1035] T bb3f4334a0554a2491987ef93a3ddb35 P e6567b3b47a2425483a479140fdad1ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "0262c1074a234ebf97c36f424ab554a2" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 43973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:59:37.297155 26122 mvcc.cc:205] Tried to move back new op lower bound from 7333119497342140416 to 7333119497233743872. Current Snapshot: MvccSnapshot[applied={T|T < 7333119497342140416}]
I20260925 04:59:37.315644 25629 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:59:37.316015 25629 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:59:37.320222 25629 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 80f9a92059b445608f2f097541ab8016: Sending DeleteTablet for 3 replicas of tablet bb3f4334a0554a2491987ef93a3ddb35
I20260925 04:59:37.321519 25754 tablet_service.cc:1514] Processing DeleteTablet for tablet bb3f4334a0554a2491987ef93a3ddb35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:59:37 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:60826
I20260925 04:59:37.322292 25888 tablet_service.cc:1514] Processing DeleteTablet for tablet bb3f4334a0554a2491987ef93a3ddb35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:59:37 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:51780
I20260925 04:59:37.322572 26022 tablet_service.cc:1514] Processing DeleteTablet for tablet bb3f4334a0554a2491987ef93a3ddb35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:59:37 UTC) from {username='kudu', principal='kudu/127.6.29.190@KRBTEST.COM'} at 127.0.0.1:54936
I20260925 04:59:37.323464 26132 tablet_replica.cc:330] T bb3f4334a0554a2491987ef93a3ddb35 P 0262c1074a234ebf97c36f424ab554a2: stopping tablet replica
I20260925 04:59:37.323975 26132 raft_consensus.cc:2229] T bb3f4334a0554a2491987ef93a3ddb35 P 0262c1074a234ebf97c36f424ab554a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:59:37.324589 26132 raft_consensus.cc:2258] T bb3f4334a0554a2491987ef93a3ddb35 P 0262c1074a234ebf97c36f424ab554a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:59:37.325709 26131 tablet_replica.cc:330] T bb3f4334a0554a2491987ef93a3ddb35 P 03d6b5ceed4f4b2bad9a0b07ccc9df44: stopping tablet replica
I20260925 04:59:37.326125 26131 raft_consensus.cc:2229] T bb3f4334a0554a2491987ef93a3ddb35 P 03d6b5ceed4f4b2bad9a0b07ccc9df44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:59:37.326313 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 25691
I20260925 04:59:37.326617 26131 raft_consensus.cc:2258] T bb3f4334a0554a2491987ef93a3ddb35 P 03d6b5ceed4f4b2bad9a0b07ccc9df44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:59:37.327003 26132 ts_tablet_manager.cc:1905] T bb3f4334a0554a2491987ef93a3ddb35 P 0262c1074a234ebf97c36f424ab554a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:59:37.333361 26131 ts_tablet_manager.cc:1905] T bb3f4334a0554a2491987ef93a3ddb35 P 03d6b5ceed4f4b2bad9a0b07ccc9df44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:59:37.335521 26132 ts_tablet_manager.cc:1918] T bb3f4334a0554a2491987ef93a3ddb35 P 0262c1074a234ebf97c36f424ab554a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:59:37.335785 26132 log.cc:1196] T bb3f4334a0554a2491987ef93a3ddb35 P 0262c1074a234ebf97c36f424ab554a2: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-2/wal/wals/bb3f4334a0554a2491987ef93a3ddb35
I20260925 04:59:37.336705 26132 ts_tablet_manager.cc:1939] T bb3f4334a0554a2491987ef93a3ddb35 P 0262c1074a234ebf97c36f424ab554a2: Deleting consensus metadata
I20260925 04:59:37.339712 25614 catalog_manager.cc:4841] TS 0262c1074a234ebf97c36f424ab554a2 (127.6.29.131:43973): tablet bb3f4334a0554a2491987ef93a3ddb35 (table test-table [id=dd73091202714cc09478bf5599c2d56e]) successfully deleted
I20260925 04:59:37.343986 26131 ts_tablet_manager.cc:1918] T bb3f4334a0554a2491987ef93a3ddb35 P 03d6b5ceed4f4b2bad9a0b07ccc9df44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:59:37.344166 26131 log.cc:1196] T bb3f4334a0554a2491987ef93a3ddb35 P 03d6b5ceed4f4b2bad9a0b07ccc9df44: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-1/wal/wals/bb3f4334a0554a2491987ef93a3ddb35
I20260925 04:59:37.344673 26130 tablet_replica.cc:330] T bb3f4334a0554a2491987ef93a3ddb35 P e6567b3b47a2425483a479140fdad1ef: stopping tablet replica
I20260925 04:59:37.344751 26131 ts_tablet_manager.cc:1939] T bb3f4334a0554a2491987ef93a3ddb35 P 03d6b5ceed4f4b2bad9a0b07ccc9df44: Deleting consensus metadata
I20260925 04:59:37.345196 26130 raft_consensus.cc:2229] T bb3f4334a0554a2491987ef93a3ddb35 P e6567b3b47a2425483a479140fdad1ef [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:59:37.346194 26130 raft_consensus.cc:2258] T bb3f4334a0554a2491987ef93a3ddb35 P e6567b3b47a2425483a479140fdad1ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:59:37.346691 25617 catalog_manager.cc:4841] TS 03d6b5ceed4f4b2bad9a0b07ccc9df44 (127.6.29.130:36287): tablet bb3f4334a0554a2491987ef93a3ddb35 (table test-table [id=dd73091202714cc09478bf5599c2d56e]) successfully deleted
I20260925 04:59:37.348960 26130 ts_tablet_manager.cc:1905] T bb3f4334a0554a2491987ef93a3ddb35 P e6567b3b47a2425483a479140fdad1ef: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 04:59:37 dist-test-slave-xzjr krb5kdc[25585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.129@KRBTEST.COM
I20260925 04:59:37.356890 26130 ts_tablet_manager.cc:1918] T bb3f4334a0554a2491987ef93a3ddb35 P e6567b3b47a2425483a479140fdad1ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:59:37.357064 26130 log.cc:1196] T bb3f4334a0554a2491987ef93a3ddb35 P e6567b3b47a2425483a479140fdad1ef: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790312053666503-6262-0/minicluster-data/ts-0/wal/wals/bb3f4334a0554a2491987ef93a3ddb35
I20260925 04:59:37.357599 26130 ts_tablet_manager.cc:1939] T bb3f4334a0554a2491987ef93a3ddb35 P e6567b3b47a2425483a479140fdad1ef: Deleting consensus metadata
I20260925 04:59:37.358043 26103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:37.327638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60850 (local address 127.6.29.129:41765)
0925 04:59:37.327833 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:37.327839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:37.339469 (+ 11630us) server_negotiation.cc:407] Connection header received
0925 04:59:37.340039 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:37.340047 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:37.340139 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:37.340233 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:37.342046 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:37.343346 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:37.345677 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:37.346590 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:37.346750 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:37.351361 (+ 4611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:37.351373 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:37.351387 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:59:37.351429 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:37.354982 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:37.355703 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:37.355710 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:37.355712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:37.355906 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:37.356306 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:37.356311 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:37.356314 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:37.356670 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:37.357178 (+ 508us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:37.357368 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 04:59:37.357611 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 04:59:37.358796 25814 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:59:37.359493 25616 catalog_manager.cc:4841] TS e6567b3b47a2425483a479140fdad1ef (127.6.29.129:41765): tablet bb3f4334a0554a2491987ef93a3ddb35 (table test-table [id=dd73091202714cc09478bf5599c2d56e]) successfully deleted
I20260925 04:59:37.465224 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 25691
I20260925 04:59:37.484130 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 25825
Sep 25 04:59:37 dist-test-slave-xzjr krb5kdc[25585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.130@KRBTEST.COM
I20260925 04:59:37.499739 26101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:37.485096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51784 (local address 127.6.29.130:36287)
0925 04:59:37.485309 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:37.485315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:37.485336 (+ 21us) server_negotiation.cc:407] Connection header received
0925 04:59:37.485409 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:37.485414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:37.485497 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:37.485607 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:37.486884 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:37.487681 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:37.489026 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:37.489435 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:37.489568 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:37.494022 (+ 4454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:37.494037 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:37.494054 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:59:37.494089 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:37.496898 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:37.497594 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:37.497603 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:37.497606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:37.497742 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:37.498138 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:37.498145 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:37.498148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:37.498543 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:37.498860 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:37.499155 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 04:59:37.499395 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 04:59:37.500578 25948 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:59:37.592095 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 25825
I20260925 04:59:37.609058 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 25959
Sep 25 04:59:37 dist-test-slave-xzjr krb5kdc[25585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.131@KRBTEST.COM
I20260925 04:59:37.623973 26105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:37.609847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54940 (local address 127.6.29.131:43973)
0925 04:59:37.610013 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:37.610017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:37.610035 (+ 18us) server_negotiation.cc:407] Connection header received
0925 04:59:37.610250 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:37.610254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:37.610319 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:37.610412 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:37.611830 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:37.612564 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:37.613895 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:37.614264 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:37.614354 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:37.618774 (+ 4420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:37.618788 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:37.618805 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:59:37.618840 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:37.621515 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:37.622111 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:37.622118 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:37.622120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:37.622274 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:37.622584 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:37.622587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:37.622589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:37.622907 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:37.623122 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:37.623460 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 04:59:37.623640 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 04:59:37.624656 26082 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:59:37.714761 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 25959
I20260925 04:59:37.731959 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 25598
I20260925 04:59:37.733151 25660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:59:37.845252 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 25598
2026-09-25T04:59:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4466 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T04:59:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:59:37Z Disabled control of system clock
I20260925 04:59:37.924000 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:33061
--webserver_interface=127.6.29.190
--webserver_port=0
--builtin_ntp_servers=127.6.29.148:42905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:33061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 04:59:38.260111 26141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:59:38.260399 26141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:59:38.260562 26141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:59:38.267697 26141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:59:38.267840 26141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:59:38.267932 26141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:59:38.268036 26141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:59:38.268115 26141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:59:38.277305 26141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:42905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:33061
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:33061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:59:38.278597 26141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:59:38.280397 26141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:59:38.292006 26149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:38.292088 26146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:38.292092 26147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:59:38.293186 26141 server_base.cc:1034] running on GCE node
I20260925 04:59:38.293918 26141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:59:38.295336 26141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:59:38.296563 26141 hybrid_clock.cc:648] HybridClock initialized: now 1790312378296505 us; error 54 us; skew 500 ppm
I20260925 04:59:38.296916 26141 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 04:59:38.299362 26141 webserver.cc:460] Webserver started at http://127.6.29.190:46671/ using document root <none> and password file <none>
I20260925 04:59:38.300164 26141 fs_manager.cc:359] Metadata directory not provided
I20260925 04:59:38.300297 26141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:59:38.300617 26141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:59:38.303601 26141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "9fbe80e490cb4b84963da73bbc9e9fe6"
format_stamp: "Formatted at 2026-09-25 04:59:38 on dist-test-slave-xzjr"
server_key: "5bc3cdd02310fce94c61f15b008129b6"
server_key_iv: "b2829dfe5f061e29754881d240ed10a7"
server_key_version: "encryptionkey@0"
I20260925 04:59:38.304500 26141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "9fbe80e490cb4b84963da73bbc9e9fe6"
format_stamp: "Formatted at 2026-09-25 04:59:38 on dist-test-slave-xzjr"
server_key: "5bc3cdd02310fce94c61f15b008129b6"
server_key_iv: "b2829dfe5f061e29754881d240ed10a7"
server_key_version: "encryptionkey@0"
I20260925 04:59:38.309571 26141 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 04:59:38.313028 26155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:38.314330 26141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260925 04:59:38.314518 26141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "9fbe80e490cb4b84963da73bbc9e9fe6"
format_stamp: "Formatted at 2026-09-25 04:59:38 on dist-test-slave-xzjr"
server_key: "5bc3cdd02310fce94c61f15b008129b6"
server_key_iv: "b2829dfe5f061e29754881d240ed10a7"
server_key_version: "encryptionkey@0"
I20260925 04:59:38.314739 26141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:59:38.328819 26141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:59:38.329669 26141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:59:38.329962 26141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:59:38.348407 26141 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:33061
I20260925 04:59:38.348430 26206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:33061 every 8 connection(s)
I20260925 04:59:38.349745 26141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:59:38.353395 26207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:59:38.359099 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 26141
I20260925 04:59:38.359292 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 04:59:38.359877 6262 external_mini_cluster.cc:1428] Setting key 71e9e7fa093ad6c3664bdb712aab039c
I20260925 04:59:38.364780 26207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fbe80e490cb4b84963da73bbc9e9fe6: Bootstrap starting.
I20260925 04:59:38.367956 26207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fbe80e490cb4b84963da73bbc9e9fe6: Neither blocks nor log segments found. Creating new log.
I20260925 04:59:38.368992 26207 log.cc:824] T 00000000000000000000000000000000 P 9fbe80e490cb4b84963da73bbc9e9fe6: Log is configured to *not* fsync() on all Append() calls
I20260925 04:59:38.372424 26207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fbe80e490cb4b84963da73bbc9e9fe6: No bootstrap required, opened a new log
I20260925 04:59:38.373579 26210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:38.361824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53716 (local address 127.6.29.190:33061)
0925 04:59:38.362876 (+ 1052us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:38.362890 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:38.362930 (+ 40us) server_negotiation.cc:407] Connection header received
0925 04:59:38.364684 (+ 1754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:38.364707 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:38.365198 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:38.365570 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:38.366142 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:38.367418 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:38.368906 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:38.369854 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:38.370067 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:38.371213 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:38.371221 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:38.371235 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:59:38.371839 (+ 604us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:38.371946 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:59:38.371969 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:38.372060 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:38.372299 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 04:59:38.372641 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":391,"threads_started":1}
I20260925 04:59:38.380649 26207 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fbe80e490cb4b84963da73bbc9e9fe6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fbe80e490cb4b84963da73bbc9e9fe6" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33061 } }
I20260925 04:59:38.381021 26207 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fbe80e490cb4b84963da73bbc9e9fe6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:59:38.381143 26207 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fbe80e490cb4b84963da73bbc9e9fe6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fbe80e490cb4b84963da73bbc9e9fe6, State: Initialized, Role: FOLLOWER
I20260925 04:59:38.381771 26207 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fbe80e490cb4b84963da73bbc9e9fe6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fbe80e490cb4b84963da73bbc9e9fe6" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33061 } }
I20260925 04:59:38.382076 26207 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fbe80e490cb4b84963da73bbc9e9fe6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:59:38.382226 26207 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fbe80e490cb4b84963da73bbc9e9fe6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:59:38.382421 26207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fbe80e490cb4b84963da73bbc9e9fe6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:59:38.384655 26207 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fbe80e490cb4b84963da73bbc9e9fe6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fbe80e490cb4b84963da73bbc9e9fe6" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33061 } }
I20260925 04:59:38.385272 26207 leader_election.cc:302] T 00000000000000000000000000000000 P 9fbe80e490cb4b84963da73bbc9e9fe6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fbe80e490cb4b84963da73bbc9e9fe6; no voters:
I20260925 04:59:38.386023 26207 leader_election.cc:288] T 00000000000000000000000000000000 P 9fbe80e490cb4b84963da73bbc9e9fe6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:59:38.386225 26212 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fbe80e490cb4b84963da73bbc9e9fe6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:59:38.386988 26212 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fbe80e490cb4b84963da73bbc9e9fe6 [term 1 LEADER]: Becoming Leader. State: Replica: 9fbe80e490cb4b84963da73bbc9e9fe6, State: Running, Role: LEADER
I20260925 04:59:38.387616 26212 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fbe80e490cb4b84963da73bbc9e9fe6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fbe80e490cb4b84963da73bbc9e9fe6" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33061 } }
I20260925 04:59:38.388268 26207 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fbe80e490cb4b84963da73bbc9e9fe6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:59:38.391007 26214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fbe80e490cb4b84963da73bbc9e9fe6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fbe80e490cb4b84963da73bbc9e9fe6. Latest consensus state: current_term: 1 leader_uuid: "9fbe80e490cb4b84963da73bbc9e9fe6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fbe80e490cb4b84963da73bbc9e9fe6" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33061 } } }
I20260925 04:59:38.391476 26214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fbe80e490cb4b84963da73bbc9e9fe6 [sys.catalog]: This master's current role is: LEADER
I20260925 04:59:38.391574 26213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fbe80e490cb4b84963da73bbc9e9fe6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fbe80e490cb4b84963da73bbc9e9fe6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fbe80e490cb4b84963da73bbc9e9fe6" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33061 } } }
I20260925 04:59:38.392704 26213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fbe80e490cb4b84963da73bbc9e9fe6 [sys.catalog]: This master's current role is: LEADER
I20260925 04:59:38.393442 26220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:59:38.398633 26220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:59:38.409494 26220 catalog_manager.cc:1347] Generated new cluster ID: 85d4f153d19f4307ba17018bcd50917c
I20260925 04:59:38.409653 26220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:59:38.440675 26220 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:59:38.442668 26220 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:59:38.459417 26220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fbe80e490cb4b84963da73bbc9e9fe6: Generated new TSK 0
I20260925 04:59:38.460616 26220 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:59:38.474099 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.129:0
--local_ip_for_outbound_sockets=127.6.29.129
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:33061
--builtin_ntp_servers=127.6.29.148:42905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 04:59:38.817327 26231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:59:38.817625 26231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:59:38.817713 26231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:59:38.825268 26231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:59:38.825376 26231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:59:38.825484 26231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.129
I20260925 04:59:38.835006 26231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:42905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.29.129
--webserver_port=0
--tserver_master_addrs=127.6.29.190:33061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.129
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:59:38.836380 26231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:59:38.838184 26231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:59:38.852172 26239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:38.852253 26237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:38.852203 26236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:59:38.857471 26231 server_base.cc:1034] running on GCE node
I20260925 04:59:38.858218 26231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:59:38.859261 26231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:59:38.860493 26231 hybrid_clock.cc:648] HybridClock initialized: now 1790312378860446 us; error 44 us; skew 500 ppm
I20260925 04:59:38.860823 26231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 04:59:38.863278 26231 webserver.cc:460] Webserver started at http://127.6.29.129:37951/ using document root <none> and password file <none>
I20260925 04:59:38.864122 26231 fs_manager.cc:359] Metadata directory not provided
I20260925 04:59:38.864256 26231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:59:38.864581 26231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:59:38.867538 26231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-0/data/instance:
uuid: "77e6140023c14ef4b5bb23623a1b0cac"
format_stamp: "Formatted at 2026-09-25 04:59:38 on dist-test-slave-xzjr"
server_key: "42d1ba358e9f59a7c93b5484b591469f"
server_key_iv: "51c644ef548127b2590eaf7a34072d1e"
server_key_version: "encryptionkey@0"
I20260925 04:59:38.868412 26231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance:
uuid: "77e6140023c14ef4b5bb23623a1b0cac"
format_stamp: "Formatted at 2026-09-25 04:59:38 on dist-test-slave-xzjr"
server_key: "42d1ba358e9f59a7c93b5484b591469f"
server_key_iv: "51c644ef548127b2590eaf7a34072d1e"
server_key_version: "encryptionkey@0"
I20260925 04:59:38.873994 26231 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 04:59:38.877768 26245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:38.879164 26231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:59:38.879357 26231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-0/wal
uuid: "77e6140023c14ef4b5bb23623a1b0cac"
format_stamp: "Formatted at 2026-09-25 04:59:38 on dist-test-slave-xzjr"
server_key: "42d1ba358e9f59a7c93b5484b591469f"
server_key_iv: "51c644ef548127b2590eaf7a34072d1e"
server_key_version: "encryptionkey@0"
I20260925 04:59:38.879595 26231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:59:38.898485 26231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:59:38.899380 26231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:59:38.899694 26231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:59:38.900751 26231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:59:38.902261 26231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:59:38.902388 26231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:38.902508 26231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:59:38.902592 26231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:38.935909 26231 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.129:43407
I20260925 04:59:38.935994 26357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.129:43407 every 8 connection(s)
I20260925 04:59:38.937318 26231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-0/data/info.pb
I20260925 04:59:38.940680 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 26231
I20260925 04:59:38.940876 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-0/wal/instance
I20260925 04:59:38.941504 6262 external_mini_cluster.cc:1428] Setting key 68fb901fa4b5738de3117eae9fbb6cb5
I20260925 04:59:38.944424 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.130:0
--local_ip_for_outbound_sockets=127.6.29.130
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:33061
--builtin_ntp_servers=127.6.29.148:42905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 04:59:38.950706 26360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:38.940035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:37467 (local address 127.6.29.190:33061)
0925 04:59:38.940556 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:38.940560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:38.942021 (+ 1461us) server_negotiation.cc:407] Connection header received
0925 04:59:38.944160 (+ 2139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:38.944165 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:38.944251 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:38.944378 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:38.945553 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:38.946287 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:38.947902 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:38.948319 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:38.948458 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:38.949837 (+ 1379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:38.949843 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:38.949848 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:59:38.949876 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:38.949917 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:59:38.949929 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:38.950022 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:38.950250 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 04:59:38.950359 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":309,"threads_started":1}
I20260925 04:59:38.951836 26361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:38.940760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:33061 (local address 127.6.29.129:37467)
0925 04:59:38.941859 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:38.941896 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:38.943893 (+ 1997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:38.944662 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:38.944675 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:38.944791 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:59:38.945387 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:38.945405 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:38.946441 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:38.946446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:38.947711 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:38.947723 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:38.948545 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:38.948554 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:38.948799 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:38.949524 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:59:38.949526 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:59:38.949568 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:59:38.949652 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:59:38.950023 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:59:38.950036 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:59:38.950046 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 04:59:38.950240 (+ 194us) client_negotiation.cc:240] Negotiation successful
0925 04:59:38.950652 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":372,"threads_started":1}
I20260925 04:59:38.953469 26358 heartbeater.cc:347] Connected to a master server at 127.6.29.190:33061
I20260925 04:59:38.953778 26358 heartbeater.cc:464] Registering TS with master...
I20260925 04:59:38.954541 26358 heartbeater.cc:511] Master 127.6.29.190:33061 requested a full tablet report, sending...
I20260925 04:59:38.956666 26172 ts_manager.cc:194] Registered new tserver with Master: 77e6140023c14ef4b5bb23623a1b0cac (127.6.29.129:43407)
I20260925 04:59:38.958180 26172 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.29.129:37467
W20260925 04:59:39.282429 26362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:59:39.282816 26362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:59:39.282938 26362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:59:39.290481 26362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:59:39.290623 26362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:59:39.290745 26362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.130
I20260925 04:59:39.300262 26362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:42905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.29.130
--webserver_port=0
--tserver_master_addrs=127.6.29.190:33061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.130
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:59:39.301559 26362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:59:39.303331 26362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:59:39.316560 26368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:39.316707 26367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:39.316682 26370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:59:39.317250 26362 server_base.cc:1034] running on GCE node
I20260925 04:59:39.317862 26362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:59:39.318912 26362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:59:39.320142 26362 hybrid_clock.cc:648] HybridClock initialized: now 1790312379320098 us; error 31 us; skew 500 ppm
I20260925 04:59:39.320487 26362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 04:59:39.322738 26362 webserver.cc:460] Webserver started at http://127.6.29.130:44405/ using document root <none> and password file <none>
I20260925 04:59:39.323451 26362 fs_manager.cc:359] Metadata directory not provided
I20260925 04:59:39.323568 26362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:59:39.323952 26362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:59:39.326861 26362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-1/data/instance:
uuid: "466df50f0a7a488bb4a0d49936497217"
format_stamp: "Formatted at 2026-09-25 04:59:39 on dist-test-slave-xzjr"
server_key: "f090f8ae391ba3d41c0fdcec4d5a2b73"
server_key_iv: "3174af2755fa13d97bdc8b3fa11305aa"
server_key_version: "encryptionkey@0"
I20260925 04:59:39.327696 26362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance:
uuid: "466df50f0a7a488bb4a0d49936497217"
format_stamp: "Formatted at 2026-09-25 04:59:39 on dist-test-slave-xzjr"
server_key: "f090f8ae391ba3d41c0fdcec4d5a2b73"
server_key_iv: "3174af2755fa13d97bdc8b3fa11305aa"
server_key_version: "encryptionkey@0"
I20260925 04:59:39.333056 26362 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 04:59:39.336802 26376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:39.338330 26362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260925 04:59:39.338526 26362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-1/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-1/wal
uuid: "466df50f0a7a488bb4a0d49936497217"
format_stamp: "Formatted at 2026-09-25 04:59:39 on dist-test-slave-xzjr"
server_key: "f090f8ae391ba3d41c0fdcec4d5a2b73"
server_key_iv: "3174af2755fa13d97bdc8b3fa11305aa"
server_key_version: "encryptionkey@0"
I20260925 04:59:39.338783 26362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:59:39.357681 26362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:59:39.358553 26362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:59:39.358884 26362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:59:39.359942 26362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:59:39.361491 26362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:59:39.361627 26362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:39.361752 26362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:59:39.361842 26362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:39.395143 26362 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.130:44651
I20260925 04:59:39.395197 26488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.130:44651 every 8 connection(s)
I20260925 04:59:39.396565 26362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-1/data/info.pb
I20260925 04:59:39.401413 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 26362
I20260925 04:59:39.401619 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-1/wal/instance
I20260925 04:59:39.402093 6262 external_mini_cluster.cc:1428] Setting key dabad284133189fe3625f6c667700159
I20260925 04:59:39.405130 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.29.131:0
--local_ip_for_outbound_sockets=127.6.29.131
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:33061
--builtin_ntp_servers=127.6.29.148:42905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 04:59:39.410260 26360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:39.399287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.130:34885 (local address 127.6.29.190:33061)
0925 04:59:39.399470 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:39.399475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:39.400990 (+ 1515us) server_negotiation.cc:407] Connection header received
0925 04:59:39.402947 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:39.402954 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:39.403074 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:39.403187 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:39.404412 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:39.405508 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:39.407059 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:39.407738 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:39.407905 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:39.409263 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:39.409272 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:39.409279 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:59:39.409304 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:39.409346 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:59:39.409364 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:39.409456 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:39.409728 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 04:59:39.409834 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 04:59:39.411202 26491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:39.399860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:33061 (local address 127.6.29.130:34885)
0925 04:59:39.400816 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:39.400854 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:39.402666 (+ 1812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:39.403386 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:39.403395 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:39.403486 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:59:39.404242 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:39.404261 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:39.405643 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:39.405647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:39.406912 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:39.406924 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:39.407972 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:39.407977 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:39.408193 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:39.408931 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:59:39.408934 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:59:39.408976 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:59:39.409069 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:59:39.409485 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:59:39.409495 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:59:39.409502 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 04:59:39.409683 (+ 181us) client_negotiation.cc:240] Negotiation successful
0925 04:59:39.410044 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":295,"threads_started":1}
I20260925 04:59:39.412863 26489 heartbeater.cc:347] Connected to a master server at 127.6.29.190:33061
I20260925 04:59:39.413259 26489 heartbeater.cc:464] Registering TS with master...
I20260925 04:59:39.414309 26489 heartbeater.cc:511] Master 127.6.29.190:33061 requested a full tablet report, sending...
I20260925 04:59:39.415966 26172 ts_manager.cc:194] Registered new tserver with Master: 466df50f0a7a488bb4a0d49936497217 (127.6.29.130:44651)
I20260925 04:59:39.417335 26172 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.29.130:34885
W20260925 04:59:39.753473 26492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:59:39.753775 26492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:59:39.753862 26492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:59:39.761169 26492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:59:39.761278 26492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:59:39.761386 26492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.29.131
I20260925 04:59:39.770809 26492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:42905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.29.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.29.131
--webserver_port=0
--tserver_master_addrs=127.6.29.190:33061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.29.131
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:59:39.772114 26492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:59:39.773874 26492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:59:39.787215 26500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:39.787261 26497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:39.787299 26498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:59:39.788004 26492 server_base.cc:1034] running on GCE node
I20260925 04:59:39.788560 26492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:59:39.789583 26492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:59:39.790800 26492 hybrid_clock.cc:648] HybridClock initialized: now 1790312379790765 us; error 29 us; skew 500 ppm
I20260925 04:59:39.791109 26492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 04:59:39.793377 26492 webserver.cc:460] Webserver started at http://127.6.29.131:36357/ using document root <none> and password file <none>
I20260925 04:59:39.794039 26492 fs_manager.cc:359] Metadata directory not provided
I20260925 04:59:39.794131 26492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:59:39.794401 26492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:59:39.797312 26492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-2/data/instance:
uuid: "ed94f3b5c6d24a1ca4870709b1391f44"
format_stamp: "Formatted at 2026-09-25 04:59:39 on dist-test-slave-xzjr"
server_key: "f6a625499f5def9064d02d192f450727"
server_key_iv: "f004b9212590802db9f8ccf0aa5d0b18"
server_key_version: "encryptionkey@0"
I20260925 04:59:39.798137 26492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance:
uuid: "ed94f3b5c6d24a1ca4870709b1391f44"
format_stamp: "Formatted at 2026-09-25 04:59:39 on dist-test-slave-xzjr"
server_key: "f6a625499f5def9064d02d192f450727"
server_key_iv: "f004b9212590802db9f8ccf0aa5d0b18"
server_key_version: "encryptionkey@0"
I20260925 04:59:39.803290 26492 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260925 04:59:39.806808 26506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:39.808202 26492 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 04:59:39.808374 26492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-2/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-2/wal
uuid: "ed94f3b5c6d24a1ca4870709b1391f44"
format_stamp: "Formatted at 2026-09-25 04:59:39 on dist-test-slave-xzjr"
server_key: "f6a625499f5def9064d02d192f450727"
server_key_iv: "f004b9212590802db9f8ccf0aa5d0b18"
server_key_version: "encryptionkey@0"
I20260925 04:59:39.808589 26492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:59:39.827466 26492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:59:39.828361 26492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:59:39.828677 26492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:59:39.829664 26492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 04:59:39.831139 26492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 04:59:39.831260 26492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:39.831405 26492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 04:59:39.831494 26492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:39.865689 26492 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.131:41159
I20260925 04:59:39.865741 26618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.131:41159 every 8 connection(s)
I20260925 04:59:39.867033 26492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-2/data/info.pb
I20260925 04:59:39.874509 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 26492
I20260925 04:59:39.874723 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-2/wal/instance
I20260925 04:59:39.875191 6262 external_mini_cluster.cc:1428] Setting key dc8c0f63b577c5ba4efa0733056f2d0d
I20260925 04:59:39.879985 26360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:39.869747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.131:40111 (local address 127.6.29.190:33061)
0925 04:59:39.869931 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:39.869936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:39.871355 (+ 1419us) server_negotiation.cc:407] Connection header received
0925 04:59:39.873183 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:39.873188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:39.873261 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:39.873353 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:39.874625 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:39.875506 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:39.877186 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:39.877667 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:39.877793 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:39.879236 (+ 1443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:39.879241 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:39.879244 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:59:39.879262 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:39.879295 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:59:39.879306 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:39.879384 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:39.879582 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 04:59:39.879666 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 04:59:39.881016 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:39.870286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:33061 (local address 127.6.29.131:40111)
0925 04:59:39.871191 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:39.871236 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:39.872918 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:39.873550 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:39.873559 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:39.873649 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0925 04:59:39.874394 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:39.874420 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:39.875679 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:39.875683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:39.877026 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:39.877040 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:39.877969 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:39.877977 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:39.878253 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:39.878912 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 04:59:39.878915 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 04:59:39.878955 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 04:59:39.879038 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 04:59:39.879399 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 04:59:39.879407 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 04:59:39.879413 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 04:59:39.879556 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 04:59:39.879958 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":284,"threads_started":1}
I20260925 04:59:39.882375 26619 heartbeater.cc:347] Connected to a master server at 127.6.29.190:33061
I20260925 04:59:39.882727 26619 heartbeater.cc:464] Registering TS with master...
I20260925 04:59:39.883636 26619 heartbeater.cc:511] Master 127.6.29.190:33061 requested a full tablet report, sending...
I20260925 04:59:39.885213 26172 ts_manager.cc:194] Registered new tserver with Master: ed94f3b5c6d24a1ca4870709b1391f44 (127.6.29.131:41159)
I20260925 04:59:39.886044 26172 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.29.131:40111
I20260925 04:59:39.890425 6262 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 04:59:39.900985 26360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:39.894885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55332 (local address 127.6.29.190:33061)
0925 04:59:39.895177 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:39.895185 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:39.895446 (+ 261us) server_negotiation.cc:407] Connection header received
0925 04:59:39.895643 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:39.895649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:39.895735 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:39.895866 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:39.896349 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:39.897049 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:39.898489 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:39.898908 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:39.899043 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:39.900269 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:39.900274 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:39.900278 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:59:39.900296 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:39.900331 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:59:39.900343 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:39.900437 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:39.900585 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 04:59:39.900685 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 04:59:39.905045 26172 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55332:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 04:59:39.912312 26172 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 04:59:39.959918 26636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:39.925579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58106 (local address 127.6.29.131:41159)
0925 04:59:39.951363 (+ 25784us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:39.951376 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:39.951424 (+ 48us) server_negotiation.cc:407] Connection header received
0925 04:59:39.951555 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:39.951579 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:39.951943 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:39.952194 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:59:39.953124 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:39.954478 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:39.957074 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:39.958632 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:39.958783 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:39.959091 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:39.959331 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 04:59:39.959506 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25578,"thread_start_us":25413,"threads_started":1}
I20260925 04:59:39.959930 26634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:39.924916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.131:41159 (local address 127.0.0.1:58106)
0925 04:59:39.950185 (+ 25269us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:39.950210 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:39.950362 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:39.952470 (+ 2108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:39.952478 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:39.952504 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:59:39.952911 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:39.952926 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:39.954613 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:39.954619 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:39.956834 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:39.956849 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:39.958864 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:39.958874 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:39.959078 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:39.959133 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 04:59:39.959274 (+ 141us) client_negotiation.cc:240] Negotiation successful
0925 04:59:39.959486 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25072,"mutex_wait_us":23,"thread_start_us":25675,"threads_started":1}
I20260925 04:59:39.960059 26631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:39.924267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.129:43407 (local address 127.0.0.1:58666)
0925 04:59:39.938949 (+ 14682us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:39.939027 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:39.939314 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:39.953566 (+ 14252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:39.953571 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:39.953597 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:59:39.953897 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:39.953908 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:39.955674 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:39.955680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:39.957430 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:39.957443 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:39.959326 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:39.959332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:39.959499 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:39.959528 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 04:59:39.959622 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 04:59:39.959755 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14284,"thread_start_us":15338,"threads_started":1}
I20260925 04:59:39.960983 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:39.924448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58666 (local address 127.6.29.129:43407)
0925 04:59:39.952946 (+ 28498us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:39.952963 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:39.952993 (+ 30us) server_negotiation.cc:407] Connection header received
0925 04:59:39.953092 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:39.953102 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:39.953345 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:39.953557 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:59:39.954084 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:39.955529 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:39.957668 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:39.958787 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:39.958915 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:39.959198 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:39.960420 (+ 1222us) server_negotiation.cc:299] Negotiation successful
0925 04:59:39.960621 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28319,"thread_start_us":28201,"threads_started":1}
I20260925 04:59:39.961423 26635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:39.925578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51780 (local address 127.6.29.130:44651)
0925 04:59:39.949372 (+ 23794us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:39.949385 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:39.951466 (+ 2081us) server_negotiation.cc:407] Connection header received
0925 04:59:39.951582 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:39.951590 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:39.951963 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:39.952181 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:59:39.953605 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:39.955260 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:39.958933 (+ 3673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:39.960392 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:39.960534 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:39.960774 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:39.960923 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 04:59:39.961078 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23569,"thread_start_us":23829,"threads_started":1}
I20260925 04:59:39.961925 26633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:39.924995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:44651 (local address 127.0.0.1:51780)
0925 04:59:39.951207 (+ 26212us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:39.951230 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:39.951344 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:39.952934 (+ 1590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:39.952939 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:39.952966 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:59:39.953412 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:39.953424 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:39.955420 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:39.955426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:39.958709 (+ 3283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:39.958726 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:39.960586 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:39.960593 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:39.960763 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:39.960795 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 04:59:39.960892 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 04:59:39.961032 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26039,"mutex_wait_us":17,"thread_start_us":25749,"threads_started":1}
I20260925 04:59:39.962253 26358 heartbeater.cc:503] Master 127.6.29.190:33061 was elected leader, sending a full tablet report...
I20260925 04:59:39.964562 26554 tablet_service.cc:1467] Processing CreateTablet for tablet 42f10160e39941a981a036cfb03b0525 (DEFAULT_TABLE table=test-table [id=936a4a7605d74b16ad3a1b1e01c78c1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:59:39.964558 26293 tablet_service.cc:1467] Processing CreateTablet for tablet 42f10160e39941a981a036cfb03b0525 (DEFAULT_TABLE table=test-table [id=936a4a7605d74b16ad3a1b1e01c78c1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:59:39.966094 26554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42f10160e39941a981a036cfb03b0525. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:59:39.966099 26293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42f10160e39941a981a036cfb03b0525. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:59:39.966107 26424 tablet_service.cc:1467] Processing CreateTablet for tablet 42f10160e39941a981a036cfb03b0525 (DEFAULT_TABLE table=test-table [id=936a4a7605d74b16ad3a1b1e01c78c1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 04:59:39.967265 26424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42f10160e39941a981a036cfb03b0525. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:59:39.978780 26637 tablet_bootstrap.cc:492] T 42f10160e39941a981a036cfb03b0525 P 466df50f0a7a488bb4a0d49936497217: Bootstrap starting.
I20260925 04:59:39.981091 26638 tablet_bootstrap.cc:492] T 42f10160e39941a981a036cfb03b0525 P 77e6140023c14ef4b5bb23623a1b0cac: Bootstrap starting.
I20260925 04:59:39.984179 26637 tablet_bootstrap.cc:654] T 42f10160e39941a981a036cfb03b0525 P 466df50f0a7a488bb4a0d49936497217: Neither blocks nor log segments found. Creating new log.
I20260925 04:59:39.984769 26638 tablet_bootstrap.cc:654] T 42f10160e39941a981a036cfb03b0525 P 77e6140023c14ef4b5bb23623a1b0cac: Neither blocks nor log segments found. Creating new log.
I20260925 04:59:39.985548 26637 log.cc:824] T 42f10160e39941a981a036cfb03b0525 P 466df50f0a7a488bb4a0d49936497217: Log is configured to *not* fsync() on all Append() calls
I20260925 04:59:39.985998 26638 log.cc:824] T 42f10160e39941a981a036cfb03b0525 P 77e6140023c14ef4b5bb23623a1b0cac: Log is configured to *not* fsync() on all Append() calls
I20260925 04:59:39.988137 26639 tablet_bootstrap.cc:492] T 42f10160e39941a981a036cfb03b0525 P ed94f3b5c6d24a1ca4870709b1391f44: Bootstrap starting.
I20260925 04:59:39.989202 26637 tablet_bootstrap.cc:492] T 42f10160e39941a981a036cfb03b0525 P 466df50f0a7a488bb4a0d49936497217: No bootstrap required, opened a new log
I20260925 04:59:39.989482 26638 tablet_bootstrap.cc:492] T 42f10160e39941a981a036cfb03b0525 P 77e6140023c14ef4b5bb23623a1b0cac: No bootstrap required, opened a new log
I20260925 04:59:39.989492 26637 ts_tablet_manager.cc:1397] T 42f10160e39941a981a036cfb03b0525 P 466df50f0a7a488bb4a0d49936497217: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260925 04:59:39.989764 26638 ts_tablet_manager.cc:1397] T 42f10160e39941a981a036cfb03b0525 P 77e6140023c14ef4b5bb23623a1b0cac: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260925 04:59:39.992012 26639 tablet_bootstrap.cc:654] T 42f10160e39941a981a036cfb03b0525 P ed94f3b5c6d24a1ca4870709b1391f44: Neither blocks nor log segments found. Creating new log.
I20260925 04:59:39.993266 26639 log.cc:824] T 42f10160e39941a981a036cfb03b0525 P ed94f3b5c6d24a1ca4870709b1391f44: Log is configured to *not* fsync() on all Append() calls
I20260925 04:59:39.996446 26639 tablet_bootstrap.cc:492] T 42f10160e39941a981a036cfb03b0525 P ed94f3b5c6d24a1ca4870709b1391f44: No bootstrap required, opened a new log
I20260925 04:59:39.996728 26639 ts_tablet_manager.cc:1397] T 42f10160e39941a981a036cfb03b0525 P ed94f3b5c6d24a1ca4870709b1391f44: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260925 04:59:39.997979 26638 raft_consensus.cc:348] T 42f10160e39941a981a036cfb03b0525 P 77e6140023c14ef4b5bb23623a1b0cac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e6140023c14ef4b5bb23623a1b0cac" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 43407 } } peers { permanent_uuid: "ed94f3b5c6d24a1ca4870709b1391f44" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 41159 } } peers { permanent_uuid: "466df50f0a7a488bb4a0d49936497217" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44651 } }
I20260925 04:59:39.998468 26638 raft_consensus.cc:374] T 42f10160e39941a981a036cfb03b0525 P 77e6140023c14ef4b5bb23623a1b0cac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:59:39.998597 26638 raft_consensus.cc:729] T 42f10160e39941a981a036cfb03b0525 P 77e6140023c14ef4b5bb23623a1b0cac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e6140023c14ef4b5bb23623a1b0cac, State: Initialized, Role: FOLLOWER
I20260925 04:59:39.999208 26638 consensus_queue.cc:260] T 42f10160e39941a981a036cfb03b0525 P 77e6140023c14ef4b5bb23623a1b0cac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e6140023c14ef4b5bb23623a1b0cac" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 43407 } } peers { permanent_uuid: "ed94f3b5c6d24a1ca4870709b1391f44" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 41159 } } peers { permanent_uuid: "466df50f0a7a488bb4a0d49936497217" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44651 } }
I20260925 04:59:39.999943 26637 raft_consensus.cc:348] T 42f10160e39941a981a036cfb03b0525 P 466df50f0a7a488bb4a0d49936497217 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e6140023c14ef4b5bb23623a1b0cac" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 43407 } } peers { permanent_uuid: "ed94f3b5c6d24a1ca4870709b1391f44" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 41159 } } peers { permanent_uuid: "466df50f0a7a488bb4a0d49936497217" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44651 } }
I20260925 04:59:40.000376 26637 raft_consensus.cc:374] T 42f10160e39941a981a036cfb03b0525 P 466df50f0a7a488bb4a0d49936497217 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:59:40.000458 26637 raft_consensus.cc:729] T 42f10160e39941a981a036cfb03b0525 P 466df50f0a7a488bb4a0d49936497217 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 466df50f0a7a488bb4a0d49936497217, State: Initialized, Role: FOLLOWER
I20260925 04:59:40.001055 26638 ts_tablet_manager.cc:1428] T 42f10160e39941a981a036cfb03b0525 P 77e6140023c14ef4b5bb23623a1b0cac: Time spent starting tablet: real 0.011s user 0.006s sys 0.006s
I20260925 04:59:40.001128 26637 consensus_queue.cc:260] T 42f10160e39941a981a036cfb03b0525 P 466df50f0a7a488bb4a0d49936497217 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e6140023c14ef4b5bb23623a1b0cac" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 43407 } } peers { permanent_uuid: "ed94f3b5c6d24a1ca4870709b1391f44" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 41159 } } peers { permanent_uuid: "466df50f0a7a488bb4a0d49936497217" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44651 } }
I20260925 04:59:40.008241 26489 heartbeater.cc:503] Master 127.6.29.190:33061 was elected leader, sending a full tablet report...
I20260925 04:59:40.009291 26637 ts_tablet_manager.cc:1428] T 42f10160e39941a981a036cfb03b0525 P 466df50f0a7a488bb4a0d49936497217: Time spent starting tablet: real 0.020s user 0.020s sys 0.000s
I20260925 04:59:40.009027 26639 raft_consensus.cc:348] T 42f10160e39941a981a036cfb03b0525 P ed94f3b5c6d24a1ca4870709b1391f44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e6140023c14ef4b5bb23623a1b0cac" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 43407 } } peers { permanent_uuid: "ed94f3b5c6d24a1ca4870709b1391f44" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 41159 } } peers { permanent_uuid: "466df50f0a7a488bb4a0d49936497217" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44651 } }
I20260925 04:59:40.009541 26639 raft_consensus.cc:374] T 42f10160e39941a981a036cfb03b0525 P ed94f3b5c6d24a1ca4870709b1391f44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:59:40.009689 26639 raft_consensus.cc:729] T 42f10160e39941a981a036cfb03b0525 P ed94f3b5c6d24a1ca4870709b1391f44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed94f3b5c6d24a1ca4870709b1391f44, State: Initialized, Role: FOLLOWER
I20260925 04:59:40.010294 26639 consensus_queue.cc:260] T 42f10160e39941a981a036cfb03b0525 P ed94f3b5c6d24a1ca4870709b1391f44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e6140023c14ef4b5bb23623a1b0cac" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 43407 } } peers { permanent_uuid: "ed94f3b5c6d24a1ca4870709b1391f44" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 41159 } } peers { permanent_uuid: "466df50f0a7a488bb4a0d49936497217" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44651 } }
I20260925 04:59:40.011397 26619 heartbeater.cc:503] Master 127.6.29.190:33061 was elected leader, sending a full tablet report...
I20260925 04:59:40.012197 26639 ts_tablet_manager.cc:1428] T 42f10160e39941a981a036cfb03b0525 P ed94f3b5c6d24a1ca4870709b1391f44: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260925 04:59:40.024732 26643 raft_consensus.cc:482] T 42f10160e39941a981a036cfb03b0525 P 77e6140023c14ef4b5bb23623a1b0cac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 04:59:40.025000 26643 raft_consensus.cc:504] T 42f10160e39941a981a036cfb03b0525 P 77e6140023c14ef4b5bb23623a1b0cac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e6140023c14ef4b5bb23623a1b0cac" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 43407 } } peers { permanent_uuid: "ed94f3b5c6d24a1ca4870709b1391f44" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 41159 } } peers { permanent_uuid: "466df50f0a7a488bb4a0d49936497217" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44651 } }
I20260925 04:59:40.026819 26643 leader_election.cc:288] T 42f10160e39941a981a036cfb03b0525 P 77e6140023c14ef4b5bb23623a1b0cac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed94f3b5c6d24a1ca4870709b1391f44 (127.6.29.131:41159), 466df50f0a7a488bb4a0d49936497217 (127.6.29.130:44651)
I20260925 04:59:40.033731 26635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:40.027322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:41989 (local address 127.6.29.130:44651)
0925 04:59:40.027531 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:40.027539 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:40.028299 (+ 760us) server_negotiation.cc:407] Connection header received
0925 04:59:40.028512 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:40.028516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:40.028586 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:40.028684 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:59:40.029322 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:40.029994 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:40.032092 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:40.032838 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:40.033026 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:40.033099 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:40.033348 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 04:59:40.033470 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 04:59:40.033773 26636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:40.027203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.29.129:56191 (local address 127.6.29.131:41159)
0925 04:59:40.027404 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:40.027412 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:40.028267 (+ 855us) server_negotiation.cc:407] Connection header received
0925 04:59:40.028486 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:40.028490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:40.028557 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:40.028653 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 04:59:40.029261 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:40.029976 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:40.032120 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:40.032896 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:40.033078 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:40.033148 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:40.033413 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 04:59:40.033550 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 04:59:40.033977 26647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:40.027181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.131:41159 (local address 127.6.29.129:56191)
0925 04:59:40.028151 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:40.028168 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:40.028342 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:40.028669 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:40.028674 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:40.028707 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:59:40.029116 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:40.029125 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:40.030105 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:40.030113 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:40.031909 (+ 1796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:40.031922 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:40.033097 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:40.033103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:40.033248 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:40.033270 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 04:59:40.033376 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 04:59:40.033498 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":829,"mutex_wait_us":106,"thread_start_us":548,"threads_started":1}
I20260925 04:59:40.034009 26646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:40.027193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.130:44651 (local address 127.6.29.129:41989)
0925 04:59:40.028190 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0925 04:59:40.028205 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 04:59:40.028370 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 04:59:40.028711 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 04:59:40.028715 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 04:59:40.028739 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 04:59:40.029145 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:40.029156 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:40.030127 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:40.030132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:40.031896 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 04:59:40.031909 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 04:59:40.033027 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 04:59:40.033033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 04:59:40.033194 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:40.033217 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 04:59:40.033321 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 04:59:40.033447 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":875,"mutex_wait_us":104,"spinlock_wait_cycles":7040,"thread_start_us":537,"threads_started":1}
I20260925 04:59:40.034957 26444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42f10160e39941a981a036cfb03b0525" candidate_uuid: "77e6140023c14ef4b5bb23623a1b0cac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "466df50f0a7a488bb4a0d49936497217" is_pre_election: true
I20260925 04:59:40.034955 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42f10160e39941a981a036cfb03b0525" candidate_uuid: "77e6140023c14ef4b5bb23623a1b0cac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed94f3b5c6d24a1ca4870709b1391f44" is_pre_election: true
I20260925 04:59:40.035451 26574 raft_consensus.cc:2454] T 42f10160e39941a981a036cfb03b0525 P ed94f3b5c6d24a1ca4870709b1391f44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e6140023c14ef4b5bb23623a1b0cac in term 0.
I20260925 04:59:40.035452 26444 raft_consensus.cc:2454] T 42f10160e39941a981a036cfb03b0525 P 466df50f0a7a488bb4a0d49936497217 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77e6140023c14ef4b5bb23623a1b0cac in term 0.
I20260925 04:59:40.036167 26248 leader_election.cc:302] T 42f10160e39941a981a036cfb03b0525 P 77e6140023c14ef4b5bb23623a1b0cac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 466df50f0a7a488bb4a0d49936497217, 77e6140023c14ef4b5bb23623a1b0cac; no voters:
I20260925 04:59:40.036571 26643 raft_consensus.cc:2781] T 42f10160e39941a981a036cfb03b0525 P 77e6140023c14ef4b5bb23623a1b0cac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 04:59:40.036715 26643 raft_consensus.cc:482] T 42f10160e39941a981a036cfb03b0525 P 77e6140023c14ef4b5bb23623a1b0cac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 04:59:40.036832 26643 raft_consensus.cc:3037] T 42f10160e39941a981a036cfb03b0525 P 77e6140023c14ef4b5bb23623a1b0cac [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:59:40.039084 26643 raft_consensus.cc:504] T 42f10160e39941a981a036cfb03b0525 P 77e6140023c14ef4b5bb23623a1b0cac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e6140023c14ef4b5bb23623a1b0cac" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 43407 } } peers { permanent_uuid: "ed94f3b5c6d24a1ca4870709b1391f44" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 41159 } } peers { permanent_uuid: "466df50f0a7a488bb4a0d49936497217" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44651 } }
I20260925 04:59:40.039803 26643 leader_election.cc:288] T 42f10160e39941a981a036cfb03b0525 P 77e6140023c14ef4b5bb23623a1b0cac [CANDIDATE]: Term 1 election: Requested vote from peers ed94f3b5c6d24a1ca4870709b1391f44 (127.6.29.131:41159), 466df50f0a7a488bb4a0d49936497217 (127.6.29.130:44651)
I20260925 04:59:40.040266 26574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42f10160e39941a981a036cfb03b0525" candidate_uuid: "77e6140023c14ef4b5bb23623a1b0cac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed94f3b5c6d24a1ca4870709b1391f44"
I20260925 04:59:40.040447 26444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42f10160e39941a981a036cfb03b0525" candidate_uuid: "77e6140023c14ef4b5bb23623a1b0cac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "466df50f0a7a488bb4a0d49936497217"
I20260925 04:59:40.040501 26574 raft_consensus.cc:3037] T 42f10160e39941a981a036cfb03b0525 P ed94f3b5c6d24a1ca4870709b1391f44 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:59:40.040668 26444 raft_consensus.cc:3037] T 42f10160e39941a981a036cfb03b0525 P 466df50f0a7a488bb4a0d49936497217 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:59:40.043105 26574 raft_consensus.cc:2454] T 42f10160e39941a981a036cfb03b0525 P ed94f3b5c6d24a1ca4870709b1391f44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e6140023c14ef4b5bb23623a1b0cac in term 1.
I20260925 04:59:40.043195 26444 raft_consensus.cc:2454] T 42f10160e39941a981a036cfb03b0525 P 466df50f0a7a488bb4a0d49936497217 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77e6140023c14ef4b5bb23623a1b0cac in term 1.
I20260925 04:59:40.043599 26247 leader_election.cc:302] T 42f10160e39941a981a036cfb03b0525 P 77e6140023c14ef4b5bb23623a1b0cac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77e6140023c14ef4b5bb23623a1b0cac, ed94f3b5c6d24a1ca4870709b1391f44; no voters:
I20260925 04:59:40.044023 26643 raft_consensus.cc:2781] T 42f10160e39941a981a036cfb03b0525 P 77e6140023c14ef4b5bb23623a1b0cac [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:59:40.044484 26643 raft_consensus.cc:686] T 42f10160e39941a981a036cfb03b0525 P 77e6140023c14ef4b5bb23623a1b0cac [term 1 LEADER]: Becoming Leader. State: Replica: 77e6140023c14ef4b5bb23623a1b0cac, State: Running, Role: LEADER
I20260925 04:59:40.045027 26643 consensus_queue.cc:237] T 42f10160e39941a981a036cfb03b0525 P 77e6140023c14ef4b5bb23623a1b0cac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e6140023c14ef4b5bb23623a1b0cac" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 43407 } } peers { permanent_uuid: "ed94f3b5c6d24a1ca4870709b1391f44" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 41159 } } peers { permanent_uuid: "466df50f0a7a488bb4a0d49936497217" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44651 } }
I20260925 04:59:40.049636 26172 catalog_manager.cc:5495] T 42f10160e39941a981a036cfb03b0525 P 77e6140023c14ef4b5bb23623a1b0cac reported cstate change: term changed from 0 to 1, leader changed from <none> to 77e6140023c14ef4b5bb23623a1b0cac (127.6.29.129). New cstate: current_term: 1 leader_uuid: "77e6140023c14ef4b5bb23623a1b0cac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77e6140023c14ef4b5bb23623a1b0cac" member_type: VOTER last_known_addr { host: "127.6.29.129" port: 43407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed94f3b5c6d24a1ca4870709b1391f44" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 41159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "466df50f0a7a488bb4a0d49936497217" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44651 } health_report { overall_health: UNKNOWN } } }
I20260925 04:59:40.077209 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:40.072034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58676 (local address 127.6.29.129:43407)
0925 04:59:40.072255 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:40.072260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:40.072280 (+ 20us) server_negotiation.cc:407] Connection header received
0925 04:59:40.072450 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:40.072453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:40.072544 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:40.072692 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 04:59:40.073013 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:40.073717 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:40.075046 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:40.075449 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:40.075545 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:40.076130 (+ 585us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 04:59:40.076658 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 04:59:40.076772 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:40.076847 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 04:59:40.076912 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 04:59:40.085871 26444 raft_consensus.cc:1261] T 42f10160e39941a981a036cfb03b0525 P 466df50f0a7a488bb4a0d49936497217 [term 1 FOLLOWER]: Refusing update from remote peer 77e6140023c14ef4b5bb23623a1b0cac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:59:40.085872 26574 raft_consensus.cc:1261] T 42f10160e39941a981a036cfb03b0525 P ed94f3b5c6d24a1ca4870709b1391f44 [term 1 FOLLOWER]: Refusing update from remote peer 77e6140023c14ef4b5bb23623a1b0cac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 04:59:40.087033 26648 consensus_queue.cc:1035] T 42f10160e39941a981a036cfb03b0525 P 77e6140023c14ef4b5bb23623a1b0cac [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed94f3b5c6d24a1ca4870709b1391f44" member_type: VOTER last_known_addr { host: "127.6.29.131" port: 41159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:59:40.087445 26643 consensus_queue.cc:1035] T 42f10160e39941a981a036cfb03b0525 P 77e6140023c14ef4b5bb23623a1b0cac [LEADER]: Connected to new peer: Peer: permanent_uuid: "466df50f0a7a488bb4a0d49936497217" member_type: VOTER last_known_addr { host: "127.6.29.130" port: 44651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 04:59:40.099630 26651 mvcc.cc:205] Tried to move back new op lower bound from 7333119508820873216 to 7333119508668301312. Current Snapshot: MvccSnapshot[applied={T|T < 7333119508820873216}]
I20260925 04:59:40.112640 26171 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:59:40.112947 26171 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 04:59:40.117200 26171 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9fbe80e490cb4b84963da73bbc9e9fe6: Sending DeleteTablet for 3 replicas of tablet 42f10160e39941a981a036cfb03b0525
I20260925 04:59:40.118642 26554 tablet_service.cc:1514] Processing DeleteTablet for tablet 42f10160e39941a981a036cfb03b0525 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:59:40 UTC) from {username='slave'} at 127.0.0.1:58106
I20260925 04:59:40.118885 26293 tablet_service.cc:1514] Processing DeleteTablet for tablet 42f10160e39941a981a036cfb03b0525 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:59:40 UTC) from {username='slave'} at 127.0.0.1:58666
I20260925 04:59:40.119403 26424 tablet_service.cc:1514] Processing DeleteTablet for tablet 42f10160e39941a981a036cfb03b0525 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 04:59:40 UTC) from {username='slave'} at 127.0.0.1:51780
I20260925 04:59:40.121253 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 26231
I20260925 04:59:40.129844 26661 tablet_replica.cc:330] T 42f10160e39941a981a036cfb03b0525 P 77e6140023c14ef4b5bb23623a1b0cac: stopping tablet replica
I20260925 04:59:40.130394 26660 tablet_replica.cc:330] T 42f10160e39941a981a036cfb03b0525 P ed94f3b5c6d24a1ca4870709b1391f44: stopping tablet replica
I20260925 04:59:40.130653 26661 raft_consensus.cc:2229] T 42f10160e39941a981a036cfb03b0525 P 77e6140023c14ef4b5bb23623a1b0cac [term 1 LEADER]: Raft consensus shutting down.
I20260925 04:59:40.130820 26660 raft_consensus.cc:2229] T 42f10160e39941a981a036cfb03b0525 P ed94f3b5c6d24a1ca4870709b1391f44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:59:40.131366 26660 raft_consensus.cc:2258] T 42f10160e39941a981a036cfb03b0525 P ed94f3b5c6d24a1ca4870709b1391f44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:59:40.131536 26661 raft_consensus.cc:2258] T 42f10160e39941a981a036cfb03b0525 P 77e6140023c14ef4b5bb23623a1b0cac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:59:40.134217 26661 ts_tablet_manager.cc:1905] T 42f10160e39941a981a036cfb03b0525 P 77e6140023c14ef4b5bb23623a1b0cac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:59:40.135007 26660 ts_tablet_manager.cc:1905] T 42f10160e39941a981a036cfb03b0525 P ed94f3b5c6d24a1ca4870709b1391f44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:59:40.142320 26661 ts_tablet_manager.cc:1918] T 42f10160e39941a981a036cfb03b0525 P 77e6140023c14ef4b5bb23623a1b0cac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:59:40.142709 26661 log.cc:1196] T 42f10160e39941a981a036cfb03b0525 P 77e6140023c14ef4b5bb23623a1b0cac: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-0/wal/wals/42f10160e39941a981a036cfb03b0525
I20260925 04:59:40.143373 26661 ts_tablet_manager.cc:1939] T 42f10160e39941a981a036cfb03b0525 P 77e6140023c14ef4b5bb23623a1b0cac: Deleting consensus metadata
I20260925 04:59:40.145546 26156 catalog_manager.cc:4841] TS 77e6140023c14ef4b5bb23623a1b0cac (127.6.29.129:43407): tablet 42f10160e39941a981a036cfb03b0525 (table test-table [id=936a4a7605d74b16ad3a1b1e01c78c1a]) successfully deleted
I20260925 04:59:40.146674 26662 tablet_replica.cc:330] T 42f10160e39941a981a036cfb03b0525 P 466df50f0a7a488bb4a0d49936497217: stopping tablet replica
I20260925 04:59:40.147171 26662 raft_consensus.cc:2229] T 42f10160e39941a981a036cfb03b0525 P 466df50f0a7a488bb4a0d49936497217 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 04:59:40.147683 26662 raft_consensus.cc:2258] T 42f10160e39941a981a036cfb03b0525 P 466df50f0a7a488bb4a0d49936497217 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 04:59:40.149441 26662 ts_tablet_manager.cc:1905] T 42f10160e39941a981a036cfb03b0525 P 466df50f0a7a488bb4a0d49936497217: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 04:59:40.150949 26660 ts_tablet_manager.cc:1918] T 42f10160e39941a981a036cfb03b0525 P ed94f3b5c6d24a1ca4870709b1391f44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:59:40.151149 26660 log.cc:1196] T 42f10160e39941a981a036cfb03b0525 P ed94f3b5c6d24a1ca4870709b1391f44: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-2/wal/wals/42f10160e39941a981a036cfb03b0525
I20260925 04:59:40.151702 26660 ts_tablet_manager.cc:1939] T 42f10160e39941a981a036cfb03b0525 P ed94f3b5c6d24a1ca4870709b1391f44: Deleting consensus metadata
I20260925 04:59:40.153728 26632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:40.122802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58688 (local address 127.6.29.129:43407)
0925 04:59:40.123033 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:40.123043 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:40.140914 (+ 17871us) server_negotiation.cc:407] Connection header received
0925 04:59:40.141431 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:40.141437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:40.141562 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:40.141634 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:40.142370 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:40.143621 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:40.145472 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:40.146189 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:40.146335 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:40.151740 (+ 5405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:40.151753 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:40.151797 (+ 44us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:59:40.152654 (+ 857us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:40.152731 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:59:40.152758 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:40.152852 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:40.153205 (+ 353us) server_negotiation.cc:299] Negotiation successful
0925 04:59:40.153333 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 04:59:40.154287 26157 catalog_manager.cc:4841] TS ed94f3b5c6d24a1ca4870709b1391f44 (127.6.29.131:41159): tablet 42f10160e39941a981a036cfb03b0525 (table test-table [id=936a4a7605d74b16ad3a1b1e01c78c1a]) successfully deleted
I20260925 04:59:40.155651 26353 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:59:40.155952 26662 ts_tablet_manager.cc:1918] T 42f10160e39941a981a036cfb03b0525 P 466df50f0a7a488bb4a0d49936497217: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 04:59:40.156131 26662 log.cc:1196] T 42f10160e39941a981a036cfb03b0525 P 466df50f0a7a488bb4a0d49936497217: Deleting WAL directory at /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790312053666503-6262-0/minicluster-data/ts-1/wal/wals/42f10160e39941a981a036cfb03b0525
I20260925 04:59:40.156687 26662 ts_tablet_manager.cc:1939] T 42f10160e39941a981a036cfb03b0525 P 466df50f0a7a488bb4a0d49936497217: Deleting consensus metadata
I20260925 04:59:40.158656 26158 catalog_manager.cc:4841] TS 466df50f0a7a488bb4a0d49936497217 (127.6.29.130:44651): tablet 42f10160e39941a981a036cfb03b0525 (table test-table [id=936a4a7605d74b16ad3a1b1e01c78c1a]) successfully deleted
I20260925 04:59:40.262352 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 26231
I20260925 04:59:40.285534 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 26362
I20260925 04:59:40.292522 26635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:40.286345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51786 (local address 127.6.29.130:44651)
0925 04:59:40.286515 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:40.286520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:40.286538 (+ 18us) server_negotiation.cc:407] Connection header received
0925 04:59:40.286713 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:40.286719 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:40.286822 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:40.286909 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:40.287290 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:40.288082 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:40.289390 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:40.289856 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:40.289989 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:40.291063 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:40.291072 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:40.291087 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:59:40.291688 (+ 601us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:40.291742 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:59:40.291810 (+ 68us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:40.291944 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:40.292078 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 04:59:40.292185 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 04:59:40.293215 26484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:59:40.386853 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 26362
I20260925 04:59:40.403592 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 26492
I20260925 04:59:40.410427 26636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:40.404409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58122 (local address 127.6.29.131:41159)
0925 04:59:40.404594 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:40.404602 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:40.404626 (+ 24us) server_negotiation.cc:407] Connection header received
0925 04:59:40.404863 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:40.404867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:40.404954 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:40.405077 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:40.405457 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:40.406191 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:40.407483 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:40.407918 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:40.408044 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:40.409148 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:40.409158 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:40.409175 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:59:40.409720 (+ 545us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:40.409768 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:59:40.409786 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:40.409913 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:40.410026 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 04:59:40.410140 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 04:59:40.411068 26614 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:59:40.501152 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 26492
I20260925 04:59:40.518227 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 26141
I20260925 04:59:40.519460 26202 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:59:40.629617 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 26141
2026-09-25T04:59:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2787 ms)
[----------] 6 tests from EncryptionPolicyTest (22124 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:59:40 dist-test-slave-xzjr krb5kdc[26670](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:59:40 dist-test-slave-xzjr krb5kdc[26670](info): set up 2 sockets
Sep 25 04:59:40 dist-test-slave-xzjr krb5kdc[26670](info): commencing operation
krb5kdc: starting...
W20260925 04:59:41.752774 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 04:59:41 dist-test-slave-xzjr krb5kdc[26670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:59:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:59:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:59:42.012109 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39917
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:59:42.350020 26683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:59:42.350328 26683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:59:42.350510 26683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:59:42.357916 26683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:59:42.358048 26683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:59:42.358150 26683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:59:42.358264 26683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:59:42.358348 26683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:59:42.368377 26683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39917
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:59:42.369709 26683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:59:42.371090 26683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:59:42.383417 26691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:42.383507 26688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:42.383510 26689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:59:42.384564 26683 server_base.cc:1034] running on GCE node
I20260925 04:59:42.385288 26683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:59:42.386698 26683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:59:42.387902 26683 hybrid_clock.cc:648] HybridClock initialized: now 1790312382387862 us; error 35 us; skew 500 ppm
I20260925 04:59:42.388283 26683 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:59:42 dist-test-slave-xzjr krb5kdc[26670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312382, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:59:42.391521 26683 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:59:42.393100 26683 webserver.cc:460] Webserver started at http://127.0.0.1:43839/ using document root <none> and password file <none>
I20260925 04:59:42.393771 26683 fs_manager.cc:359] Metadata directory not provided
I20260925 04:59:42.393872 26683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:59:42.394153 26683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:59:42.397166 26683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "624598168fb947a0b3ed5029cd0a218b"
format_stamp: "Formatted at 2026-09-25 04:59:42 on dist-test-slave-xzjr"
server_key: "a6853db6e6ec9077d0add19677e7fe15"
server_key_iv: "a7e75d44bebd704c1f053ac738ddff2a"
server_key_version: "encryptionkey@0"
I20260925 04:59:42.397897 26683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "624598168fb947a0b3ed5029cd0a218b"
format_stamp: "Formatted at 2026-09-25 04:59:42 on dist-test-slave-xzjr"
server_key: "a6853db6e6ec9077d0add19677e7fe15"
server_key_iv: "a7e75d44bebd704c1f053ac738ddff2a"
server_key_version: "encryptionkey@0"
I20260925 04:59:42.403174 26683 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 04:59:42.406713 26698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:42.408197 26683 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:59:42.408401 26683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "624598168fb947a0b3ed5029cd0a218b"
format_stamp: "Formatted at 2026-09-25 04:59:42 on dist-test-slave-xzjr"
server_key: "a6853db6e6ec9077d0add19677e7fe15"
server_key_iv: "a7e75d44bebd704c1f053ac738ddff2a"
server_key_version: "encryptionkey@0"
I20260925 04:59:42.408603 26683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:59:42.443648 26683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:59:42.448453 26683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:59:42.448760 26683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:59:42.467703 26683 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39917
I20260925 04:59:42.467700 26749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39917 every 8 connection(s)
I20260925 04:59:42.469287 26683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:59:42.473263 26750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:59:42.478201 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 26683
I20260925 04:59:42.478454 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 04:59:42.479099 6262 external_mini_cluster.cc:1428] Setting key 8caf179cccc6ba5dfa87fbbc5dcdd43f
I20260925 04:59:42.485065 26750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 624598168fb947a0b3ed5029cd0a218b: Bootstrap starting.
I20260925 04:59:42.489055 26750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 624598168fb947a0b3ed5029cd0a218b: Neither blocks nor log segments found. Creating new log.
I20260925 04:59:42.490374 26750 log.cc:824] T 00000000000000000000000000000000 P 624598168fb947a0b3ed5029cd0a218b: Log is configured to *not* fsync() on all Append() calls
I20260925 04:59:42.494343 26750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 624598168fb947a0b3ed5029cd0a218b: No bootstrap required, opened a new log
Sep 25 04:59:42 dist-test-slave-xzjr krb5kdc[26670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 04:59:42.505143 26753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:42.481112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34263 (local address 127.0.0.1:39917)
0925 04:59:42.482012 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:42.482028 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:42.482072 (+ 44us) server_negotiation.cc:407] Connection header received
0925 04:59:42.483990 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:42.484017 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:42.484518 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:42.484875 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:42.486634 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:42.488092 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:42.489952 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:42.490778 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:42.491023 (+ 245us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:42.495987 (+ 4964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:42.496003 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:42.496023 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:59:42.496083 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:42.500243 (+ 4160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:42.501050 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:42.501059 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:42.501070 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:42.501297 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:42.501717 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:42.501722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:42.501726 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:42.502502 (+ 776us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:42.502820 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:42.503251 (+ 431us) server_negotiation.cc:299] Negotiation successful
0925 04:59:42.503625 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":426,"threads_started":1}
I20260925 04:59:42.505623 26750 raft_consensus.cc:348] T 00000000000000000000000000000000 P 624598168fb947a0b3ed5029cd0a218b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "624598168fb947a0b3ed5029cd0a218b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39917 } }
I20260925 04:59:42.506023 26750 raft_consensus.cc:374] T 00000000000000000000000000000000 P 624598168fb947a0b3ed5029cd0a218b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:59:42.506150 26750 raft_consensus.cc:729] T 00000000000000000000000000000000 P 624598168fb947a0b3ed5029cd0a218b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 624598168fb947a0b3ed5029cd0a218b, State: Initialized, Role: FOLLOWER
I20260925 04:59:42.506740 26750 consensus_queue.cc:260] T 00000000000000000000000000000000 P 624598168fb947a0b3ed5029cd0a218b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "624598168fb947a0b3ed5029cd0a218b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39917 } }
I20260925 04:59:42.507030 26750 raft_consensus.cc:388] T 00000000000000000000000000000000 P 624598168fb947a0b3ed5029cd0a218b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:59:42.507256 26750 raft_consensus.cc:482] T 00000000000000000000000000000000 P 624598168fb947a0b3ed5029cd0a218b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:59:42.507452 26750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 624598168fb947a0b3ed5029cd0a218b [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:59:42.509673 6262 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:59:42.510270 26750 raft_consensus.cc:504] T 00000000000000000000000000000000 P 624598168fb947a0b3ed5029cd0a218b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "624598168fb947a0b3ed5029cd0a218b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39917 } }
I20260925 04:59:42.510912 26750 leader_election.cc:302] T 00000000000000000000000000000000 P 624598168fb947a0b3ed5029cd0a218b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 624598168fb947a0b3ed5029cd0a218b; no voters:
I20260925 04:59:42.511601 26750 leader_election.cc:288] T 00000000000000000000000000000000 P 624598168fb947a0b3ed5029cd0a218b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:59:42.511827 26755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 624598168fb947a0b3ed5029cd0a218b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:59:42.514321 26750 sys_catalog.cc:564] T 00000000000000000000000000000000 P 624598168fb947a0b3ed5029cd0a218b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:59:42.519224 26755 raft_consensus.cc:686] T 00000000000000000000000000000000 P 624598168fb947a0b3ed5029cd0a218b [term 1 LEADER]: Becoming Leader. State: Replica: 624598168fb947a0b3ed5029cd0a218b, State: Running, Role: LEADER
I20260925 04:59:42.520022 26755 consensus_queue.cc:237] T 00000000000000000000000000000000 P 624598168fb947a0b3ed5029cd0a218b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "624598168fb947a0b3ed5029cd0a218b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39917 } }
I20260925 04:59:42.523087 26756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 624598168fb947a0b3ed5029cd0a218b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "624598168fb947a0b3ed5029cd0a218b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "624598168fb947a0b3ed5029cd0a218b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39917 } } }
I20260925 04:59:42.523468 26756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 624598168fb947a0b3ed5029cd0a218b [sys.catalog]: This master's current role is: LEADER
I20260925 04:59:42.523989 26757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 624598168fb947a0b3ed5029cd0a218b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 624598168fb947a0b3ed5029cd0a218b. Latest consensus state: current_term: 1 leader_uuid: "624598168fb947a0b3ed5029cd0a218b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "624598168fb947a0b3ed5029cd0a218b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39917 } } }
I20260925 04:59:42.524283 26757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 624598168fb947a0b3ed5029cd0a218b [sys.catalog]: This master's current role is: LEADER
I20260925 04:59:42.526607 26767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:59:42.540208 26767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:59:42.542665 26753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:42.520077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58823 (local address 127.0.0.1:39917)
0925 04:59:42.520301 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:42.520311 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:42.520594 (+ 283us) server_negotiation.cc:407] Connection header received
0925 04:59:42.520839 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:42.520844 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:42.520926 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:42.521044 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:42.524174 (+ 3130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:42.525642 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:42.527638 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:42.528340 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:42.528475 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:42.533262 (+ 4787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:42.533273 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:42.533281 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:59:42.533341 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:42.538051 (+ 4710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:42.539189 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:42.539196 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:42.539200 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:42.539350 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:42.539970 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:42.539976 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:42.539980 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:42.540523 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:42.540788 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:42.542041 (+ 1253us) server_negotiation.cc:299] Negotiation successful
0925 04:59:42.542240 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 04:59:42.551134 26767 catalog_manager.cc:1347] Generated new cluster ID: 108ddc20325f4d8188a59d587376cfda
I20260925 04:59:42.551270 26767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:59:42.578233 26767 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:59:42.579550 26767 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:59:42.591167 26767 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 624598168fb947a0b3ed5029cd0a218b: Generated new TSK 0
I20260925 04:59:42.592140 26767 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:59:42.602952 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 26683
I20260925 04:59:42.604329 26745 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:59:42.699374 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 26683
W20260925 04:59:42.717454 26668 connection.cc:535] client connection to 127.0.0.1:39917 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39917): Connection reset by peer (error 104)
2026-09-25T04:59:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2061 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:59:42 dist-test-slave-xzjr krb5kdc[26783](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:59:42 dist-test-slave-xzjr krb5kdc[26783](info): set up 2 sockets
Sep 25 04:59:42 dist-test-slave-xzjr krb5kdc[26783](info): commencing operation
krb5kdc: starting...
W20260925 04:59:43.804661 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 04:59:43 dist-test-slave-xzjr krb5kdc[26783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:59:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:59:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:59:44.071048 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35325
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:59:44.417192 26796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:59:44.417490 26796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:59:44.417585 26796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:59:44.424873 26796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:59:44.424994 26796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:59:44.425041 26796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:59:44.425107 26796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:59:44.425149 26796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:59:44.435024 26796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35325
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:59:44.436374 26796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:59:44.437630 26796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:59:44.449692 26802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:44.449666 26801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:44.449676 26804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:59:44.450357 26796 server_base.cc:1034] running on GCE node
I20260925 04:59:44.451241 26796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:59:44.452793 26796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:59:44.454010 26796 hybrid_clock.cc:648] HybridClock initialized: now 1790312384453927 us; error 76 us; skew 500 ppm
I20260925 04:59:44.454391 26796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:59:44 dist-test-slave-xzjr krb5kdc[26783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:59:44.457849 26796 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:59:44.459446 26796 webserver.cc:460] Webserver started at http://127.0.0.1:45997/ using document root <none> and password file <none>
I20260925 04:59:44.460238 26796 fs_manager.cc:359] Metadata directory not provided
I20260925 04:59:44.460371 26796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:59:44.460692 26796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:59:44.463865 26796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "55030e7850824384be284c3104da180e"
format_stamp: "Formatted at 2026-09-25 04:59:44 on dist-test-slave-xzjr"
server_key: "e90257b2e4ed7b5f3b8ab34907f9c833"
server_key_iv: "1817259fa9645d2bde3baa6afa0c5ac0"
server_key_version: "encryptionkey@0"
I20260925 04:59:44.464700 26796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "55030e7850824384be284c3104da180e"
format_stamp: "Formatted at 2026-09-25 04:59:44 on dist-test-slave-xzjr"
server_key: "e90257b2e4ed7b5f3b8ab34907f9c833"
server_key_iv: "1817259fa9645d2bde3baa6afa0c5ac0"
server_key_version: "encryptionkey@0"
I20260925 04:59:44.470060 26796 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 04:59:44.473713 26811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:44.475117 26796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 04:59:44.475353 26796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "55030e7850824384be284c3104da180e"
format_stamp: "Formatted at 2026-09-25 04:59:44 on dist-test-slave-xzjr"
server_key: "e90257b2e4ed7b5f3b8ab34907f9c833"
server_key_iv: "1817259fa9645d2bde3baa6afa0c5ac0"
server_key_version: "encryptionkey@0"
I20260925 04:59:44.475596 26796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:59:44.497360 26796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:59:44.501760 26796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:59:44.502071 26796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:59:44.521936 26796 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35325
I20260925 04:59:44.521977 26862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35325 every 8 connection(s)
I20260925 04:59:44.523415 26796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:59:44.526652 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 26796
I20260925 04:59:44.526877 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 04:59:44.527348 6262 external_mini_cluster.cc:1428] Setting key c3287d98cec7517511a099632dd3e219
I20260925 04:59:44.527979 26863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:59:44.542601 26866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:44.529406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54031 (local address 127.0.0.1:35325)
0925 04:59:44.530292 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:44.530304 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:44.530347 (+ 43us) server_negotiation.cc:407] Connection header received
0925 04:59:44.532887 (+ 2540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:44.532914 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:44.533430 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:44.533801 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:44.534466 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:44.535925 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:44.537807 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:44.538545 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:44.538678 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:44.540107 (+ 1429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:44.540116 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:44.540134 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:59:44.540745 (+ 611us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:44.540828 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:59:44.540854 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:44.540979 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:44.541314 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 04:59:44.541519 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":390,"threads_started":1}
I20260925 04:59:44.543011 26863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55030e7850824384be284c3104da180e: Bootstrap starting.
I20260925 04:59:44.546558 26863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55030e7850824384be284c3104da180e: Neither blocks nor log segments found. Creating new log.
I20260925 04:59:44.546784 6262 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:59:44.547698 26863 log.cc:824] T 00000000000000000000000000000000 P 55030e7850824384be284c3104da180e: Log is configured to *not* fsync() on all Append() calls
I20260925 04:59:44.551069 26863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55030e7850824384be284c3104da180e: No bootstrap required, opened a new log
I20260925 04:59:44.558540 26866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:44.551637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60771 (local address 127.0.0.1:35325)
0925 04:59:44.551886 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:44.551894 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:44.552207 (+ 313us) server_negotiation.cc:407] Connection header received
0925 04:59:44.552411 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:44.552415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:44.552474 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:44.552551 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:44.553202 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:44.553970 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:44.555867 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:44.556302 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:44.556409 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:44.557656 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:44.557661 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:44.557664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:59:44.557685 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:44.557731 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:59:44.557745 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:44.557833 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:44.558106 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 04:59:44.558175 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260925 04:59:44.561095 26863 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55030e7850824384be284c3104da180e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55030e7850824384be284c3104da180e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35325 } }
I20260925 04:59:44.561605 26863 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55030e7850824384be284c3104da180e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:59:44.561736 26863 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55030e7850824384be284c3104da180e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55030e7850824384be284c3104da180e, State: Initialized, Role: FOLLOWER
I20260925 04:59:44.562330 26863 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55030e7850824384be284c3104da180e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55030e7850824384be284c3104da180e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35325 } }
I20260925 04:59:44.562582 26863 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55030e7850824384be284c3104da180e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:59:44.563011 26863 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55030e7850824384be284c3104da180e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:59:44.563264 26863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55030e7850824384be284c3104da180e [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:59:44.565623 26863 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55030e7850824384be284c3104da180e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55030e7850824384be284c3104da180e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35325 } }
I20260925 04:59:44.566182 26863 leader_election.cc:302] T 00000000000000000000000000000000 P 55030e7850824384be284c3104da180e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55030e7850824384be284c3104da180e; no voters:
I20260925 04:59:44.566811 26863 leader_election.cc:288] T 00000000000000000000000000000000 P 55030e7850824384be284c3104da180e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:59:44.566965 26873 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55030e7850824384be284c3104da180e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:59:44.567783 26873 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55030e7850824384be284c3104da180e [term 1 LEADER]: Becoming Leader. State: Replica: 55030e7850824384be284c3104da180e, State: Running, Role: LEADER
I20260925 04:59:44.568347 26873 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55030e7850824384be284c3104da180e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55030e7850824384be284c3104da180e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35325 } }
I20260925 04:59:44.569110 26863 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55030e7850824384be284c3104da180e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:59:44.571197 26874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55030e7850824384be284c3104da180e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55030e7850824384be284c3104da180e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55030e7850824384be284c3104da180e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35325 } } }
I20260925 04:59:44.571593 26874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55030e7850824384be284c3104da180e [sys.catalog]: This master's current role is: LEADER
I20260925 04:59:44.572073 26875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55030e7850824384be284c3104da180e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55030e7850824384be284c3104da180e. Latest consensus state: current_term: 1 leader_uuid: "55030e7850824384be284c3104da180e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55030e7850824384be284c3104da180e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35325 } } }
I20260925 04:59:44.572379 26875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55030e7850824384be284c3104da180e [sys.catalog]: This master's current role is: LEADER
I20260925 04:59:44.574335 26879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:59:44.579027 26879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:59:44.588120 26879 catalog_manager.cc:1347] Generated new cluster ID: 15841bec0b1e4e8a807b8acf2dc5f040
I20260925 04:59:44.588248 26879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:59:44.610962 26879 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:59:44.612428 26879 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:59:44.623471 26879 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55030e7850824384be284c3104da180e: Generated new TSK 0
I20260925 04:59:44.624442 26879 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:59:44.630627 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 26796
W20260925 04:59:44.631671 26858 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54031
W20260925 04:59:44.632519 6262 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 26796: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 04:59:44.632718 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 26796
W20260925 04:59:44.650125 26781 connection.cc:535] client connection to 127.0.0.1:35325 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35325): Connection reset by peer (error 104)
2026-09-25T04:59:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1932 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:59:44 dist-test-slave-xzjr krb5kdc[26894](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:59:44 dist-test-slave-xzjr krb5kdc[26894](info): set up 2 sockets
Sep 25 04:59:44 dist-test-slave-xzjr krb5kdc[26894](info): commencing operation
krb5kdc: starting...
W20260925 04:59:45.744587 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 04:59:45 dist-test-slave-xzjr krb5kdc[26894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:59:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:59:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:59:46.003535 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39103
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39103
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:59:46.336041 26907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:59:46.336329 26907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:59:46.336534 26907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:59:46.343781 26907 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:59:46.343897 26907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:59:46.344023 26907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:59:46.344135 26907 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:59:46.344200 26907 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:59:46.354115 26907 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39103
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:59:46.355512 26907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:59:46.356890 26907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:59:46.368891 26912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:46.368916 26913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:46.369189 26915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:59:46.369783 26907 server_base.cc:1034] running on GCE node
I20260925 04:59:46.370690 26907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:59:46.372227 26907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:59:46.373478 26907 hybrid_clock.cc:648] HybridClock initialized: now 1790312386373430 us; error 41 us; skew 500 ppm
I20260925 04:59:46.373874 26907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:59:46 dist-test-slave-xzjr krb5kdc[26894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312386, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:59:46.383843 26907 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:59:46.385507 26907 webserver.cc:460] Webserver started at http://127.0.0.1:38263/ using document root <none> and password file <none>
I20260925 04:59:46.386240 26907 fs_manager.cc:359] Metadata directory not provided
I20260925 04:59:46.386384 26907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:59:46.386703 26907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:59:46.389727 26907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "37268e3d78d44e42848f23f415ab6779"
format_stamp: "Formatted at 2026-09-25 04:59:46 on dist-test-slave-xzjr"
server_key: "3fc72f2b7122b2b665c8a61387d6a462"
server_key_iv: "4765936ae16f6658fb7acc6fd4e189ad"
server_key_version: "encryptionkey@0"
I20260925 04:59:46.390479 26907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "37268e3d78d44e42848f23f415ab6779"
format_stamp: "Formatted at 2026-09-25 04:59:46 on dist-test-slave-xzjr"
server_key: "3fc72f2b7122b2b665c8a61387d6a462"
server_key_iv: "4765936ae16f6658fb7acc6fd4e189ad"
server_key_version: "encryptionkey@0"
I20260925 04:59:46.395694 26907 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 04:59:46.399128 26922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:46.400640 26907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 04:59:46.400822 26907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "37268e3d78d44e42848f23f415ab6779"
format_stamp: "Formatted at 2026-09-25 04:59:46 on dist-test-slave-xzjr"
server_key: "3fc72f2b7122b2b665c8a61387d6a462"
server_key_iv: "4765936ae16f6658fb7acc6fd4e189ad"
server_key_version: "encryptionkey@0"
I20260925 04:59:46.401016 26907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:59:46.406755 26907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:59:46.410672 26907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:59:46.410955 26907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:59:46.429294 26907 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39103
I20260925 04:59:46.429334 26973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39103 every 8 connection(s)
I20260925 04:59:46.430708 26907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:59:46.434465 26974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:59:46.437407 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 26907
I20260925 04:59:46.437598 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 04:59:46.438159 6262 external_mini_cluster.cc:1428] Setting key 15ed05015b08989c4fe28c39adfc8e48
I20260925 04:59:46.445794 26977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:46.440038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49715 (local address 127.0.0.1:39103)
0925 04:59:46.440798 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:46.440810 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:46.440850 (+ 40us) server_negotiation.cc:407] Connection header received
0925 04:59:46.442487 (+ 1637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:46.442509 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:46.442827 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:46.443235 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:46.444276 (+ 1041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:46.444282 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:46.444294 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:59:46.444313 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:46.444367 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:59:46.444388 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:46.444639 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:46.444768 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 04:59:46.444946 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":316,"threads_started":1}
I20260925 04:59:46.446568 26974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37268e3d78d44e42848f23f415ab6779: Bootstrap starting.
I20260925 04:59:46.449555 6262 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:59:46.450064 26974 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37268e3d78d44e42848f23f415ab6779: Neither blocks nor log segments found. Creating new log.
I20260925 04:59:46.451062 26974 log.cc:824] T 00000000000000000000000000000000 P 37268e3d78d44e42848f23f415ab6779: Log is configured to *not* fsync() on all Append() calls
I20260925 04:59:46.454519 26974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37268e3d78d44e42848f23f415ab6779: No bootstrap required, opened a new log
I20260925 04:59:46.456737 26977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:46.454148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37329 (local address 127.0.0.1:39103)
0925 04:59:46.454375 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:46.454382 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:46.454695 (+ 313us) server_negotiation.cc:407] Connection header received
0925 04:59:46.454880 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:46.454883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:46.454934 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:46.455046 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:46.456047 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:46.456052 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:46.456055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:59:46.456070 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:46.456143 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:59:46.456159 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:46.456298 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:46.456387 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 04:59:46.456466 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260925 04:59:46.463508 26974 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37268e3d78d44e42848f23f415ab6779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37268e3d78d44e42848f23f415ab6779" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39103 } }
I20260925 04:59:46.464027 26974 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37268e3d78d44e42848f23f415ab6779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:59:46.464144 26974 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37268e3d78d44e42848f23f415ab6779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37268e3d78d44e42848f23f415ab6779, State: Initialized, Role: FOLLOWER
I20260925 04:59:46.464754 26974 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37268e3d78d44e42848f23f415ab6779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37268e3d78d44e42848f23f415ab6779" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39103 } }
I20260925 04:59:46.465049 26974 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37268e3d78d44e42848f23f415ab6779 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:59:46.465200 26974 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37268e3d78d44e42848f23f415ab6779 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:59:46.465390 26974 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37268e3d78d44e42848f23f415ab6779 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:59:46.468142 26974 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37268e3d78d44e42848f23f415ab6779 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37268e3d78d44e42848f23f415ab6779" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39103 } }
I20260925 04:59:46.468690 26974 leader_election.cc:302] T 00000000000000000000000000000000 P 37268e3d78d44e42848f23f415ab6779 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37268e3d78d44e42848f23f415ab6779; no voters:
I20260925 04:59:46.469372 26974 leader_election.cc:288] T 00000000000000000000000000000000 P 37268e3d78d44e42848f23f415ab6779 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:59:46.469524 26984 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37268e3d78d44e42848f23f415ab6779 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:59:46.470220 26984 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37268e3d78d44e42848f23f415ab6779 [term 1 LEADER]: Becoming Leader. State: Replica: 37268e3d78d44e42848f23f415ab6779, State: Running, Role: LEADER
I20260925 04:59:46.470932 26984 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37268e3d78d44e42848f23f415ab6779 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37268e3d78d44e42848f23f415ab6779" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39103 } }
I20260925 04:59:46.471729 26974 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37268e3d78d44e42848f23f415ab6779 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:59:46.473939 26985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37268e3d78d44e42848f23f415ab6779 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37268e3d78d44e42848f23f415ab6779" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37268e3d78d44e42848f23f415ab6779" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39103 } } }
I20260925 04:59:46.474128 26986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37268e3d78d44e42848f23f415ab6779 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37268e3d78d44e42848f23f415ab6779. Latest consensus state: current_term: 1 leader_uuid: "37268e3d78d44e42848f23f415ab6779" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37268e3d78d44e42848f23f415ab6779" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39103 } } }
I20260925 04:59:46.474395 26986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37268e3d78d44e42848f23f415ab6779 [sys.catalog]: This master's current role is: LEADER
I20260925 04:59:46.474395 26985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37268e3d78d44e42848f23f415ab6779 [sys.catalog]: This master's current role is: LEADER
I20260925 04:59:46.476402 26992 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:59:46.481355 26992 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:59:46.490916 26992 catalog_manager.cc:1347] Generated new cluster ID: eb2a1a2ed5684ac5bbf2cd557b44d3b6
I20260925 04:59:46.491050 26992 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:59:46.512414 26992 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:59:46.512905 26992 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:59:46.523591 26992 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37268e3d78d44e42848f23f415ab6779: Generated new TSK 0
I20260925 04:59:46.524605 26992 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:59:46.536860 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 26907
W20260925 04:59:46.538087 26969 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49715
W20260925 04:59:46.538859 6262 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 26907: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 04:59:46.539026 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 26907
2026-09-25T04:59:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1903 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:59:46 dist-test-slave-xzjr krb5kdc[27005](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:59:46 dist-test-slave-xzjr krb5kdc[27005](info): set up 2 sockets
Sep 25 04:59:46 dist-test-slave-xzjr krb5kdc[27005](info): commencing operation
krb5kdc: starting...
W20260925 04:59:47.648556 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 04:59:47 dist-test-slave-xzjr krb5kdc[27005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:59:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:59:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:59:47.915223 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32847
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:59:48.251895 27018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:59:48.252203 27018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:59:48.252368 27018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:59:48.259912 27018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:59:48.260058 27018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:59:48.260195 27018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:59:48.260294 27018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:59:48.260360 27018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:59:48.270519 27018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:59:48.271859 27018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:59:48.273164 27018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:59:48.284895 27026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:48.284969 27023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:48.284969 27024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:59:48.286010 27018 server_base.cc:1034] running on GCE node
I20260925 04:59:48.286759 27018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:59:48.288201 27018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:59:48.289463 27018 hybrid_clock.cc:648] HybridClock initialized: now 1790312388289404 us; error 47 us; skew 500 ppm
I20260925 04:59:48.289855 27018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:59:48 dist-test-slave-xzjr krb5kdc[27005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:59:48.293460 27018 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:59:48.294802 27018 webserver.cc:460] Webserver started at http://127.0.0.1:39659/ using document root <none> and password file <none>
I20260925 04:59:48.295429 27018 fs_manager.cc:359] Metadata directory not provided
I20260925 04:59:48.295526 27018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:59:48.295811 27018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:59:48.298887 27018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "770fb45c55434357a575b7ffda2a4a89"
format_stamp: "Formatted at 2026-09-25 04:59:48 on dist-test-slave-xzjr"
server_key: "f4ffb88f7a1069ef9bd4b472b6044312"
server_key_iv: "3817f0d58533efc4cee790d2a0733f33"
server_key_version: "encryptionkey@0"
I20260925 04:59:48.299569 27018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "770fb45c55434357a575b7ffda2a4a89"
format_stamp: "Formatted at 2026-09-25 04:59:48 on dist-test-slave-xzjr"
server_key: "f4ffb88f7a1069ef9bd4b472b6044312"
server_key_iv: "3817f0d58533efc4cee790d2a0733f33"
server_key_version: "encryptionkey@0"
I20260925 04:59:48.304580 27018 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260925 04:59:48.307924 27033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:48.309330 27018 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 04:59:48.309521 27018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "770fb45c55434357a575b7ffda2a4a89"
format_stamp: "Formatted at 2026-09-25 04:59:48 on dist-test-slave-xzjr"
server_key: "f4ffb88f7a1069ef9bd4b472b6044312"
server_key_iv: "3817f0d58533efc4cee790d2a0733f33"
server_key_version: "encryptionkey@0"
I20260925 04:59:48.309690 27018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:59:48.328291 27018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:59:48.332692 27018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:59:48.332989 27018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:59:48.351006 27018 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32847
I20260925 04:59:48.351030 27084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32847 every 8 connection(s)
I20260925 04:59:48.352419 27018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:59:48.356324 27085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:59:48.360080 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 27018
I20260925 04:59:48.360265 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 04:59:48.360742 6262 external_mini_cluster.cc:1428] Setting key ded592a5503a43c5b1fe9e589c2e6938
I20260925 04:59:48.369280 27085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 770fb45c55434357a575b7ffda2a4a89: Bootstrap starting.
I20260925 04:59:48.373276 27085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 770fb45c55434357a575b7ffda2a4a89: Neither blocks nor log segments found. Creating new log.
I20260925 04:59:48.374643 27085 log.cc:824] T 00000000000000000000000000000000 P 770fb45c55434357a575b7ffda2a4a89: Log is configured to *not* fsync() on all Append() calls
Sep 25 04:59:48 dist-test-slave-xzjr krb5kdc[27005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 04:59:48.378150 27085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 770fb45c55434357a575b7ffda2a4a89: No bootstrap required, opened a new log
I20260925 04:59:48.384963 27088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:48.362473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55991 (local address 127.0.0.1:32847)
0925 04:59:48.363231 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:48.363243 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:48.363285 (+ 42us) server_negotiation.cc:407] Connection header received
0925 04:59:48.364982 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:48.365005 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:48.365445 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:48.365770 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:48.367612 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:48.368994 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:48.370809 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:48.371494 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:48.371621 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:48.376465 (+ 4844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:48.376476 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:48.376497 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:59:48.376538 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:48.380506 (+ 3968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:48.381178 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:48.381192 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:48.381202 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:48.381413 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:48.381755 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:48.381760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:48.381763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:48.382480 (+ 717us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:48.382802 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:48.383078 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 04:59:48.383508 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":341,"threads_started":1}
I20260925 04:59:48.388968 6262 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:59:48.388765 27085 raft_consensus.cc:348] T 00000000000000000000000000000000 P 770fb45c55434357a575b7ffda2a4a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "770fb45c55434357a575b7ffda2a4a89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32847 } }
I20260925 04:59:48.389215 27085 raft_consensus.cc:374] T 00000000000000000000000000000000 P 770fb45c55434357a575b7ffda2a4a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:59:48.389310 27085 raft_consensus.cc:729] T 00000000000000000000000000000000 P 770fb45c55434357a575b7ffda2a4a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 770fb45c55434357a575b7ffda2a4a89, State: Initialized, Role: FOLLOWER
I20260925 04:59:48.389927 27085 consensus_queue.cc:260] T 00000000000000000000000000000000 P 770fb45c55434357a575b7ffda2a4a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "770fb45c55434357a575b7ffda2a4a89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32847 } }
I20260925 04:59:48.390210 27085 raft_consensus.cc:388] T 00000000000000000000000000000000 P 770fb45c55434357a575b7ffda2a4a89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:59:48.390331 27085 raft_consensus.cc:482] T 00000000000000000000000000000000 P 770fb45c55434357a575b7ffda2a4a89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:59:48.390487 27085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 770fb45c55434357a575b7ffda2a4a89 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:59:48.393801 27085 raft_consensus.cc:504] T 00000000000000000000000000000000 P 770fb45c55434357a575b7ffda2a4a89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "770fb45c55434357a575b7ffda2a4a89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32847 } }
I20260925 04:59:48.394439 27085 leader_election.cc:302] T 00000000000000000000000000000000 P 770fb45c55434357a575b7ffda2a4a89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 770fb45c55434357a575b7ffda2a4a89; no voters:
I20260925 04:59:48.395110 27085 leader_election.cc:288] T 00000000000000000000000000000000 P 770fb45c55434357a575b7ffda2a4a89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:59:48.395259 27095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 770fb45c55434357a575b7ffda2a4a89 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:59:48.397225 27085 sys_catalog.cc:564] T 00000000000000000000000000000000 P 770fb45c55434357a575b7ffda2a4a89 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:59:48.402115 27095 raft_consensus.cc:686] T 00000000000000000000000000000000 P 770fb45c55434357a575b7ffda2a4a89 [term 1 LEADER]: Becoming Leader. State: Replica: 770fb45c55434357a575b7ffda2a4a89, State: Running, Role: LEADER
I20260925 04:59:48.402654 27095 consensus_queue.cc:237] T 00000000000000000000000000000000 P 770fb45c55434357a575b7ffda2a4a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "770fb45c55434357a575b7ffda2a4a89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32847 } }
I20260925 04:59:48.405601 27096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 770fb45c55434357a575b7ffda2a4a89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "770fb45c55434357a575b7ffda2a4a89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "770fb45c55434357a575b7ffda2a4a89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32847 } } }
I20260925 04:59:48.405979 27096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 770fb45c55434357a575b7ffda2a4a89 [sys.catalog]: This master's current role is: LEADER
I20260925 04:59:48.408740 27097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 770fb45c55434357a575b7ffda2a4a89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 770fb45c55434357a575b7ffda2a4a89. Latest consensus state: current_term: 1 leader_uuid: "770fb45c55434357a575b7ffda2a4a89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "770fb45c55434357a575b7ffda2a4a89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32847 } } }
I20260925 04:59:48.412880 27097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 770fb45c55434357a575b7ffda2a4a89 [sys.catalog]: This master's current role is: LEADER
I20260925 04:59:48.415287 27088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:48.394114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45939 (local address 127.0.0.1:32847)
0925 04:59:48.394291 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:48.394296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:48.394806 (+ 510us) server_negotiation.cc:407] Connection header received
0925 04:59:48.395026 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:48.395036 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:48.395124 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:48.395231 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:48.397632 (+ 2401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:48.399212 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:48.401693 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:48.402684 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:48.402855 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:48.406208 (+ 3353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:48.406217 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:48.406224 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:59:48.406276 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:48.410748 (+ 4472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:48.412165 (+ 1417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:48.412173 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:48.412180 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:48.412348 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:48.413152 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:48.413160 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:48.413167 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:48.413694 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:48.413976 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:48.414660 (+ 684us) server_negotiation.cc:299] Negotiation successful
0925 04:59:48.414934 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 04:59:48.415721 27104 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:59:48.421116 27104 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:59:48.429630 27104 catalog_manager.cc:1347] Generated new cluster ID: 8628f98f6fd04089ab27e94e79ddc8b4
I20260925 04:59:48.429739 27104 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:59:48.441063 27104 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:59:48.442957 27104 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:59:48.454329 27104 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 770fb45c55434357a575b7ffda2a4a89: Generated new TSK 0
I20260925 04:59:48.455278 27104 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:59:48.471962 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 27018
I20260925 04:59:48.473039 27080 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:59:48.565677 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 27018
2026-09-25T04:59:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2028 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:59:48 dist-test-slave-xzjr krb5kdc[27118](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:59:48 dist-test-slave-xzjr krb5kdc[27118](info): set up 2 sockets
Sep 25 04:59:48 dist-test-slave-xzjr krb5kdc[27118](info): commencing operation
krb5kdc: starting...
W20260925 04:59:49.673847 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 04:59:49 dist-test-slave-xzjr krb5kdc[27118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:59:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:59:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:59:49.948161 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42815
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:59:50.290469 27131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:59:50.290818 27131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:59:50.290990 27131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:59:50.298328 27131 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:59:50.298482 27131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:59:50.298599 27131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:59:50.298702 27131 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:59:50.298792 27131 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:59:50.309135 27131 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:59:50.310526 27131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:59:50.311879 27131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:59:50.323895 27136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:50.323997 27137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:50.323973 27139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:59:50.324577 27131 server_base.cc:1034] running on GCE node
I20260925 04:59:50.325670 27131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:59:50.327191 27131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:59:50.328438 27131 hybrid_clock.cc:648] HybridClock initialized: now 1790312390328370 us; error 53 us; skew 500 ppm
I20260925 04:59:50.328814 27131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:59:50 dist-test-slave-xzjr krb5kdc[27118](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:59:50.332041 27131 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:59:50.333619 27131 webserver.cc:460] Webserver started at http://127.0.0.1:34441/ using document root <none> and password file <none>
I20260925 04:59:50.334444 27131 fs_manager.cc:359] Metadata directory not provided
I20260925 04:59:50.334623 27131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:59:50.334957 27131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:59:50.338090 27131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "bc16392c663a4626b8decbb192797d68"
format_stamp: "Formatted at 2026-09-25 04:59:50 on dist-test-slave-xzjr"
server_key: "32258d7fabc2144646f2ed2661e692e4"
server_key_iv: "7a5f70c3a736794678d4bd217bb935ea"
server_key_version: "encryptionkey@0"
I20260925 04:59:50.338865 27131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "bc16392c663a4626b8decbb192797d68"
format_stamp: "Formatted at 2026-09-25 04:59:50 on dist-test-slave-xzjr"
server_key: "32258d7fabc2144646f2ed2661e692e4"
server_key_iv: "7a5f70c3a736794678d4bd217bb935ea"
server_key_version: "encryptionkey@0"
I20260925 04:59:50.344148 27131 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 04:59:50.347544 27146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:50.349063 27131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:59:50.349282 27131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "bc16392c663a4626b8decbb192797d68"
format_stamp: "Formatted at 2026-09-25 04:59:50 on dist-test-slave-xzjr"
server_key: "32258d7fabc2144646f2ed2661e692e4"
server_key_iv: "7a5f70c3a736794678d4bd217bb935ea"
server_key_version: "encryptionkey@0"
I20260925 04:59:50.349501 27131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:59:50.366415 27131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:59:50.370628 27131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:59:50.370944 27131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:59:50.389611 27131 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42815
I20260925 04:59:50.389655 27197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42815 every 8 connection(s)
I20260925 04:59:50.391054 27131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:59:50.394960 27198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:59:50.395220 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 27131
I20260925 04:59:50.395416 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 04:59:50.396023 6262 external_mini_cluster.cc:1428] Setting key 180fa75581e83e6c6cd8c70c4bccb8ce
I20260925 04:59:50.408699 27198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc16392c663a4626b8decbb192797d68: Bootstrap starting.
I20260925 04:59:50.409498 27201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:50.397930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56859 (local address 127.0.0.1:42815)
0925 04:59:50.398782 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:50.398797 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:50.398842 (+ 45us) server_negotiation.cc:407] Connection header received
0925 04:59:50.400486 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:50.400509 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:50.401002 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:50.401440 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:50.402029 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:50.403443 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:50.404854 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:50.405576 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:50.405804 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:50.407058 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:50.407066 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:50.407080 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:59:50.407635 (+ 555us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:50.407718 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:59:50.407744 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:50.407922 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:50.408139 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 04:59:50.408440 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":374,"threads_started":1}
I20260925 04:59:50.411934 27198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc16392c663a4626b8decbb192797d68: Neither blocks nor log segments found. Creating new log.
I20260925 04:59:50.413129 27198 log.cc:824] T 00000000000000000000000000000000 P bc16392c663a4626b8decbb192797d68: Log is configured to *not* fsync() on all Append() calls
I20260925 04:59:50.413254 6262 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:59:50.415946 27198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc16392c663a4626b8decbb192797d68: No bootstrap required, opened a new log
I20260925 04:59:50.424109 27198 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc16392c663a4626b8decbb192797d68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc16392c663a4626b8decbb192797d68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42815 } }
I20260925 04:59:50.424585 27198 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc16392c663a4626b8decbb192797d68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:59:50.424705 27198 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc16392c663a4626b8decbb192797d68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc16392c663a4626b8decbb192797d68, State: Initialized, Role: FOLLOWER
I20260925 04:59:50.425307 27198 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc16392c663a4626b8decbb192797d68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc16392c663a4626b8decbb192797d68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42815 } }
I20260925 04:59:50.425557 27198 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc16392c663a4626b8decbb192797d68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:59:50.425735 27198 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc16392c663a4626b8decbb192797d68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:59:50.425927 27198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc16392c663a4626b8decbb192797d68 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:59:50.428452 27201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:50.421343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51475 (local address 127.0.0.1:42815)
0925 04:59:50.421545 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:50.421555 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:50.422062 (+ 507us) server_negotiation.cc:407] Connection header received
0925 04:59:50.422239 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:50.422245 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:50.422339 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:50.422456 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:50.422941 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:50.424021 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:50.425658 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:50.426386 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:50.426513 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:50.427616 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:50.427624 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:50.427633 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:59:50.427655 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:50.427702 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:59:50.427722 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:50.427862 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:50.428023 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 04:59:50.428122 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 04:59:50.429127 27198 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc16392c663a4626b8decbb192797d68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc16392c663a4626b8decbb192797d68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42815 } }
I20260925 04:59:50.429694 27198 leader_election.cc:302] T 00000000000000000000000000000000 P bc16392c663a4626b8decbb192797d68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc16392c663a4626b8decbb192797d68; no voters:
I20260925 04:59:50.430287 27198 leader_election.cc:288] T 00000000000000000000000000000000 P bc16392c663a4626b8decbb192797d68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:59:50.430428 27208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc16392c663a4626b8decbb192797d68 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:59:50.431144 27208 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc16392c663a4626b8decbb192797d68 [term 1 LEADER]: Becoming Leader. State: Replica: bc16392c663a4626b8decbb192797d68, State: Running, Role: LEADER
I20260925 04:59:50.431675 27208 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc16392c663a4626b8decbb192797d68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc16392c663a4626b8decbb192797d68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42815 } }
I20260925 04:59:50.432621 27198 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc16392c663a4626b8decbb192797d68 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:59:50.435134 27210 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc16392c663a4626b8decbb192797d68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc16392c663a4626b8decbb192797d68. Latest consensus state: current_term: 1 leader_uuid: "bc16392c663a4626b8decbb192797d68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc16392c663a4626b8decbb192797d68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42815 } } }
I20260925 04:59:50.435112 27209 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc16392c663a4626b8decbb192797d68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc16392c663a4626b8decbb192797d68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc16392c663a4626b8decbb192797d68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42815 } } }
I20260925 04:59:50.435562 27210 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc16392c663a4626b8decbb192797d68 [sys.catalog]: This master's current role is: LEADER
I20260925 04:59:50.435581 27209 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc16392c663a4626b8decbb192797d68 [sys.catalog]: This master's current role is: LEADER
I20260925 04:59:50.437076 27214 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:59:50.442754 27214 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:59:50.455214 27214 catalog_manager.cc:1347] Generated new cluster ID: 54158139e4884e90bec7809b3be59caa
I20260925 04:59:50.455399 27214 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:59:50.482134 27214 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:59:50.484248 27214 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:59:50.502171 27214 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc16392c663a4626b8decbb192797d68: Generated new TSK 0
I20260925 04:59:50.503299 27214 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:59:50.519940 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 27131
W20260925 04:59:50.521092 27193 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56859
W20260925 04:59:50.521842 6262 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 27131: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 04:59:50.522002 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 27131
2026-09-25T04:59:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1953 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:59:50 dist-test-slave-xzjr krb5kdc[27229](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:59:50 dist-test-slave-xzjr krb5kdc[27229](info): set up 2 sockets
Sep 25 04:59:50 dist-test-slave-xzjr krb5kdc[27229](info): commencing operation
krb5kdc: starting...
W20260925 04:59:51.620707 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 04:59:51 dist-test-slave-xzjr krb5kdc[27229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312391, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:59:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:59:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:59:51.887163 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40269
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:59:52.229674 27242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:59:52.229969 27242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:59:52.230151 27242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:59:52.237720 27242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:59:52.237845 27242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:59:52.237955 27242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:59:52.238078 27242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:59:52.238178 27242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:59:52.248483 27242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:59:52.249914 27242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:59:52.251314 27242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:59:52.264313 27247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:52.264369 27248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:52.264564 27250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:59:52.265317 27242 server_base.cc:1034] running on GCE node
I20260925 04:59:52.266155 27242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:59:52.267604 27242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:59:52.268823 27242 hybrid_clock.cc:648] HybridClock initialized: now 1790312392268776 us; error 34 us; skew 500 ppm
I20260925 04:59:52.269157 27242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:59:52 dist-test-slave-xzjr krb5kdc[27229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:59:52.276048 27242 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:59:52.278045 27242 webserver.cc:460] Webserver started at http://127.0.0.1:46715/ using document root <none> and password file <none>
I20260925 04:59:52.279019 27242 fs_manager.cc:359] Metadata directory not provided
I20260925 04:59:52.279186 27242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:59:52.279520 27242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:59:52.282691 27242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "003db43d53ea4768a804cd5ef9c798b2"
format_stamp: "Formatted at 2026-09-25 04:59:52 on dist-test-slave-xzjr"
server_key: "71c75af323913d88c9760c5851fe51ec"
server_key_iv: "efb38d0f4c416c19501e4035ec050b4c"
server_key_version: "encryptionkey@0"
I20260925 04:59:52.283488 27242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "003db43d53ea4768a804cd5ef9c798b2"
format_stamp: "Formatted at 2026-09-25 04:59:52 on dist-test-slave-xzjr"
server_key: "71c75af323913d88c9760c5851fe51ec"
server_key_iv: "efb38d0f4c416c19501e4035ec050b4c"
server_key_version: "encryptionkey@0"
I20260925 04:59:52.289008 27242 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260925 04:59:52.292780 27257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:52.294281 27242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:59:52.294492 27242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "003db43d53ea4768a804cd5ef9c798b2"
format_stamp: "Formatted at 2026-09-25 04:59:52 on dist-test-slave-xzjr"
server_key: "71c75af323913d88c9760c5851fe51ec"
server_key_iv: "efb38d0f4c416c19501e4035ec050b4c"
server_key_version: "encryptionkey@0"
I20260925 04:59:52.294713 27242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:59:52.300952 27242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:59:52.305526 27242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:59:52.305851 27242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:59:52.324733 27242 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40269
I20260925 04:59:52.324781 27308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40269 every 8 connection(s)
I20260925 04:59:52.326174 27242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:59:52.329972 27309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:59:52.332942 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 27242
I20260925 04:59:52.333117 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 04:59:52.333683 6262 external_mini_cluster.cc:1428] Setting key 5bed70d909bb17a2e35c26727bd47bc6
I20260925 04:59:52.341611 27312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:52.335585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45707 (local address 127.0.0.1:40269)
0925 04:59:52.336389 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:52.336401 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:52.336445 (+ 44us) server_negotiation.cc:407] Connection header received
0925 04:59:52.338101 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:52.338125 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:52.338485 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:52.338936 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:52.340005 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:52.340011 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:52.340024 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:59:52.340041 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:52.340099 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:59:52.340122 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:52.340375 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:52.340519 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 04:59:52.340712 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":327,"threads_started":1}
I20260925 04:59:52.342862 27309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 003db43d53ea4768a804cd5ef9c798b2: Bootstrap starting.
I20260925 04:59:52.345459 6262 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:59:52.346374 27309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 003db43d53ea4768a804cd5ef9c798b2: Neither blocks nor log segments found. Creating new log.
I20260925 04:59:52.347401 27309 log.cc:824] T 00000000000000000000000000000000 P 003db43d53ea4768a804cd5ef9c798b2: Log is configured to *not* fsync() on all Append() calls
I20260925 04:59:52.350592 27309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 003db43d53ea4768a804cd5ef9c798b2: No bootstrap required, opened a new log
I20260925 04:59:52.352512 27312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:52.349927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53947 (local address 127.0.0.1:40269)
0925 04:59:52.350143 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:52.350150 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:52.350470 (+ 320us) server_negotiation.cc:407] Connection header received
0925 04:59:52.350679 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:52.350684 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:52.350753 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:52.350836 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:52.351848 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:52.351852 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:52.351855 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:59:52.351869 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:52.351907 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:59:52.351921 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:52.352010 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:52.352160 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 04:59:52.352213 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 04:59:52.359081 27309 raft_consensus.cc:348] T 00000000000000000000000000000000 P 003db43d53ea4768a804cd5ef9c798b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "003db43d53ea4768a804cd5ef9c798b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40269 } }
I20260925 04:59:52.359604 27309 raft_consensus.cc:374] T 00000000000000000000000000000000 P 003db43d53ea4768a804cd5ef9c798b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:59:52.359735 27309 raft_consensus.cc:729] T 00000000000000000000000000000000 P 003db43d53ea4768a804cd5ef9c798b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 003db43d53ea4768a804cd5ef9c798b2, State: Initialized, Role: FOLLOWER
I20260925 04:59:52.360380 27309 consensus_queue.cc:260] T 00000000000000000000000000000000 P 003db43d53ea4768a804cd5ef9c798b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "003db43d53ea4768a804cd5ef9c798b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40269 } }
I20260925 04:59:52.360646 27309 raft_consensus.cc:388] T 00000000000000000000000000000000 P 003db43d53ea4768a804cd5ef9c798b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:59:52.360800 27309 raft_consensus.cc:482] T 00000000000000000000000000000000 P 003db43d53ea4768a804cd5ef9c798b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:59:52.360987 27309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 003db43d53ea4768a804cd5ef9c798b2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:59:52.363533 27309 raft_consensus.cc:504] T 00000000000000000000000000000000 P 003db43d53ea4768a804cd5ef9c798b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "003db43d53ea4768a804cd5ef9c798b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40269 } }
I20260925 04:59:52.364130 27309 leader_election.cc:302] T 00000000000000000000000000000000 P 003db43d53ea4768a804cd5ef9c798b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 003db43d53ea4768a804cd5ef9c798b2; no voters:
I20260925 04:59:52.364748 27309 leader_election.cc:288] T 00000000000000000000000000000000 P 003db43d53ea4768a804cd5ef9c798b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:59:52.364897 27319 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 003db43d53ea4768a804cd5ef9c798b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:59:52.365594 27319 raft_consensus.cc:686] T 00000000000000000000000000000000 P 003db43d53ea4768a804cd5ef9c798b2 [term 1 LEADER]: Becoming Leader. State: Replica: 003db43d53ea4768a804cd5ef9c798b2, State: Running, Role: LEADER
I20260925 04:59:52.366194 27319 consensus_queue.cc:237] T 00000000000000000000000000000000 P 003db43d53ea4768a804cd5ef9c798b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "003db43d53ea4768a804cd5ef9c798b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40269 } }
I20260925 04:59:52.366946 27309 sys_catalog.cc:564] T 00000000000000000000000000000000 P 003db43d53ea4768a804cd5ef9c798b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:59:52.368930 27321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 003db43d53ea4768a804cd5ef9c798b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 003db43d53ea4768a804cd5ef9c798b2. Latest consensus state: current_term: 1 leader_uuid: "003db43d53ea4768a804cd5ef9c798b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "003db43d53ea4768a804cd5ef9c798b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40269 } } }
I20260925 04:59:52.369355 27321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 003db43d53ea4768a804cd5ef9c798b2 [sys.catalog]: This master's current role is: LEADER
I20260925 04:59:52.370332 27326 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:59:52.370179 27320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 003db43d53ea4768a804cd5ef9c798b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "003db43d53ea4768a804cd5ef9c798b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "003db43d53ea4768a804cd5ef9c798b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40269 } } }
I20260925 04:59:52.370978 27320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 003db43d53ea4768a804cd5ef9c798b2 [sys.catalog]: This master's current role is: LEADER
I20260925 04:59:52.377274 27326 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:59:52.387756 27326 catalog_manager.cc:1347] Generated new cluster ID: a552f9a9bfb8401a82a8f1fe5379ad02
I20260925 04:59:52.387912 27326 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:59:52.404943 27326 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:59:52.405485 27326 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:59:52.416767 27326 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 003db43d53ea4768a804cd5ef9c798b2: Generated new TSK 0
I20260925 04:59:52.417945 27326 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:59:52.434928 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 27242
W20260925 04:59:52.436172 27304 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45707
W20260925 04:59:52.436906 6262 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 27242: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 04:59:52.437067 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 27242
2026-09-25T04:59:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1914 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:59:52 dist-test-slave-xzjr krb5kdc[27340](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:59:52 dist-test-slave-xzjr krb5kdc[27340](info): set up 2 sockets
Sep 25 04:59:52 dist-test-slave-xzjr krb5kdc[27340](info): commencing operation
krb5kdc: starting...
W20260925 04:59:53.536579 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 04:59:53 dist-test-slave-xzjr krb5kdc[27340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:59:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:59:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:59:53.803944 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35489
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:59:54.142059 27353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:59:54.142379 27353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:59:54.142547 27353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:59:54.149947 27353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:59:54.150079 27353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:59:54.150192 27353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:59:54.150321 27353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:59:54.150408 27353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:59:54.160682 27353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:59:54.162096 27353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:59:54.163476 27353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:59:54.175727 27358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:54.175694 27361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:54.175647 27359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:59:54.176520 27353 server_base.cc:1034] running on GCE node
I20260925 04:59:54.177301 27353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:59:54.178759 27353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:59:54.179986 27353 hybrid_clock.cc:648] HybridClock initialized: now 1790312394179940 us; error 67 us; skew 500 ppm
I20260925 04:59:54.180368 27353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:59:54 dist-test-slave-xzjr krb5kdc[27340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312394, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:59:54.183512 27353 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:59:54.185088 27353 webserver.cc:460] Webserver started at http://127.0.0.1:43413/ using document root <none> and password file <none>
I20260925 04:59:54.185811 27353 fs_manager.cc:359] Metadata directory not provided
I20260925 04:59:54.185971 27353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:59:54.186293 27353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:59:54.189433 27353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "1f050b38ce954912a84c56cf490c5077"
format_stamp: "Formatted at 2026-09-25 04:59:54 on dist-test-slave-xzjr"
server_key: "d2329ebe5f5fe926957f953207578613"
server_key_iv: "f79bf7942fdce6c76822b9dcca989202"
server_key_version: "encryptionkey@0"
I20260925 04:59:54.190214 27353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "1f050b38ce954912a84c56cf490c5077"
format_stamp: "Formatted at 2026-09-25 04:59:54 on dist-test-slave-xzjr"
server_key: "d2329ebe5f5fe926957f953207578613"
server_key_iv: "f79bf7942fdce6c76822b9dcca989202"
server_key_version: "encryptionkey@0"
I20260925 04:59:54.195915 27353 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 04:59:54.199414 27368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:54.201015 27353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:59:54.201228 27353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "1f050b38ce954912a84c56cf490c5077"
format_stamp: "Formatted at 2026-09-25 04:59:54 on dist-test-slave-xzjr"
server_key: "d2329ebe5f5fe926957f953207578613"
server_key_iv: "f79bf7942fdce6c76822b9dcca989202"
server_key_version: "encryptionkey@0"
I20260925 04:59:54.201438 27353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:59:54.216245 27353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:59:54.222437 27353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:59:54.222849 27353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:59:54.242353 27353 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35489
I20260925 04:59:54.242413 27419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35489 every 8 connection(s)
I20260925 04:59:54.243841 27353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:59:54.247736 27420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:59:54.249583 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 27353
I20260925 04:59:54.249765 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 04:59:54.250237 6262 external_mini_cluster.cc:1428] Setting key f818b4947575c30cbf55bf182d7dac39
I20260925 04:59:54.263423 27420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f050b38ce954912a84c56cf490c5077: Bootstrap starting.
Sep 25 04:59:54 dist-test-slave-xzjr krb5kdc[27340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 04:59:54.266965 27420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f050b38ce954912a84c56cf490c5077: Neither blocks nor log segments found. Creating new log.
I20260925 04:59:54.268553 27420 log.cc:824] T 00000000000000000000000000000000 P 1f050b38ce954912a84c56cf490c5077: Log is configured to *not* fsync() on all Append() calls
I20260925 04:59:54.272117 27420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f050b38ce954912a84c56cf490c5077: No bootstrap required, opened a new log
I20260925 04:59:54.275626 27423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:54.252194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.182:34419 (local address 127.0.0.1:35489)
0925 04:59:54.253039 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:54.253056 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:54.253115 (+ 59us) server_negotiation.cc:407] Connection header received
0925 04:59:54.255076 (+ 1961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:54.255107 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:54.255695 (+ 588us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:54.256196 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:54.257441 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:54.259641 (+ 2200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:54.261045 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:54.262186 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:54.262390 (+ 204us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:54.266368 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:54.266379 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:54.266399 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:59:54.266451 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:54.270604 (+ 4153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:54.271514 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:54.271523 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:54.271532 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:54.271795 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:54.272265 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:54.272270 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:54.272274 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:54.273039 (+ 765us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:54.273388 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:54.273750 (+ 362us) server_negotiation.cc:299] Negotiation successful
0925 04:59:54.274110 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":362,"threads_started":1}
I20260925 04:59:54.279927 6262 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:59:54.283792 27420 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f050b38ce954912a84c56cf490c5077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f050b38ce954912a84c56cf490c5077" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35489 } }
I20260925 04:59:54.284507 27420 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f050b38ce954912a84c56cf490c5077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:59:54.284649 27420 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f050b38ce954912a84c56cf490c5077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f050b38ce954912a84c56cf490c5077, State: Initialized, Role: FOLLOWER
I20260925 04:59:54.285611 27420 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f050b38ce954912a84c56cf490c5077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f050b38ce954912a84c56cf490c5077" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35489 } }
I20260925 04:59:54.286012 27420 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f050b38ce954912a84c56cf490c5077 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:59:54.286177 27420 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f050b38ce954912a84c56cf490c5077 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:59:54.286353 27420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f050b38ce954912a84c56cf490c5077 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:59:54.289414 27420 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f050b38ce954912a84c56cf490c5077 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f050b38ce954912a84c56cf490c5077" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35489 } }
I20260925 04:59:54.289945 27420 leader_election.cc:302] T 00000000000000000000000000000000 P 1f050b38ce954912a84c56cf490c5077 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f050b38ce954912a84c56cf490c5077; no voters:
I20260925 04:59:54.290697 27420 leader_election.cc:288] T 00000000000000000000000000000000 P 1f050b38ce954912a84c56cf490c5077 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:59:54.290817 27430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f050b38ce954912a84c56cf490c5077 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:59:54.292402 27430 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f050b38ce954912a84c56cf490c5077 [term 1 LEADER]: Becoming Leader. State: Replica: 1f050b38ce954912a84c56cf490c5077, State: Running, Role: LEADER
I20260925 04:59:54.292991 27430 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f050b38ce954912a84c56cf490c5077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f050b38ce954912a84c56cf490c5077" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35489 } }
I20260925 04:59:54.293301 27420 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f050b38ce954912a84c56cf490c5077 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:59:54.295719 27432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f050b38ce954912a84c56cf490c5077 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f050b38ce954912a84c56cf490c5077. Latest consensus state: current_term: 1 leader_uuid: "1f050b38ce954912a84c56cf490c5077" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f050b38ce954912a84c56cf490c5077" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35489 } } }
I20260925 04:59:54.295938 27431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f050b38ce954912a84c56cf490c5077 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f050b38ce954912a84c56cf490c5077" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f050b38ce954912a84c56cf490c5077" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35489 } } }
I20260925 04:59:54.296176 27432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f050b38ce954912a84c56cf490c5077 [sys.catalog]: This master's current role is: LEADER
I20260925 04:59:54.296209 27431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f050b38ce954912a84c56cf490c5077 [sys.catalog]: This master's current role is: LEADER
I20260925 04:59:54.297020 27435 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:59:54.310789 27423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:54.287421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.182:37581 (local address 127.0.0.1:35489)
0925 04:59:54.287701 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:54.287710 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:54.288309 (+ 599us) server_negotiation.cc:407] Connection header received
0925 04:59:54.288452 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:54.288459 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:54.288557 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:54.288682 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:54.290097 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:54.291497 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:54.293521 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:54.294288 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:54.294439 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:54.299079 (+ 4640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:54.299091 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:54.299099 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 04:59:54.299158 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:54.304481 (+ 5323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:54.306244 (+ 1763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:54.306254 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:54.306264 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:54.306469 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 04:59:54.307282 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 04:59:54.307291 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 04:59:54.307297 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 04:59:54.307921 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:54.309628 (+ 1707us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:54.309886 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 04:59:54.310158 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 04:59:54.317384 27435 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:59:54.326215 27435 catalog_manager.cc:1347] Generated new cluster ID: 49aede2f6a074635ab347b4e60f5d76b
I20260925 04:59:54.326347 27435 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:59:54.343155 27435 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:59:54.344537 27435 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:59:54.352244 27435 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f050b38ce954912a84c56cf490c5077: Generated new TSK 0
I20260925 04:59:54.353062 27435 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:59:54.362807 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 27353
I20260925 04:59:54.364362 27415 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 04:59:54.457137 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 27353
W20260925 04:59:54.473513 27338 connection.cc:535] client connection to 127.0.0.1:35489 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35489): Connection reset by peer (error 104)
2026-09-25T04:59:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2022 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:59:54 dist-test-slave-xzjr krb5kdc[27452](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:59:54 dist-test-slave-xzjr krb5kdc[27452](info): set up 2 sockets
Sep 25 04:59:54 dist-test-slave-xzjr krb5kdc[27452](info): commencing operation
krb5kdc: starting...
W20260925 04:59:55.556655 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 04:59:55 dist-test-slave-xzjr krb5kdc[27452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:59:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:59:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:59:55.819540 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41679
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:59:56.149477 27465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:59:56.149857 27465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:59:56.150044 27465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:59:56.157826 27465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:59:56.158017 27465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:59:56.158129 27465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:59:56.158245 27465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:59:56.158325 27465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:59:56.168344 27465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:59:56.169729 27465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:59:56.171087 27465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:59:56.183097 27470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:56.183136 27471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:56.183193 27473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:59:56.184448 27465 server_base.cc:1034] running on GCE node
I20260925 04:59:56.185227 27465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:59:56.186664 27465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:59:56.187863 27465 hybrid_clock.cc:648] HybridClock initialized: now 1790312396187823 us; error 59 us; skew 500 ppm
I20260925 04:59:56.188212 27465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:59:56 dist-test-slave-xzjr krb5kdc[27452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312396, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:59:56.191244 27465 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:59:56.192795 27465 webserver.cc:460] Webserver started at http://127.0.0.1:36161/ using document root <none> and password file <none>
I20260925 04:59:56.193495 27465 fs_manager.cc:359] Metadata directory not provided
I20260925 04:59:56.193648 27465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:59:56.193966 27465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:59:56.197026 27465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "059a762968d54aa381ae04ec3fe086dc"
format_stamp: "Formatted at 2026-09-25 04:59:56 on dist-test-slave-xzjr"
server_key: "3806550d39f166a0edfcc7fde74afc7a"
server_key_iv: "ef2626108c82e8fbaa407bc8e18e81ac"
server_key_version: "encryptionkey@0"
I20260925 04:59:56.197808 27465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "059a762968d54aa381ae04ec3fe086dc"
format_stamp: "Formatted at 2026-09-25 04:59:56 on dist-test-slave-xzjr"
server_key: "3806550d39f166a0edfcc7fde74afc7a"
server_key_iv: "ef2626108c82e8fbaa407bc8e18e81ac"
server_key_version: "encryptionkey@0"
I20260925 04:59:56.203275 27465 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 04:59:56.206983 27480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:56.208413 27465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 04:59:56.208603 27465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "059a762968d54aa381ae04ec3fe086dc"
format_stamp: "Formatted at 2026-09-25 04:59:56 on dist-test-slave-xzjr"
server_key: "3806550d39f166a0edfcc7fde74afc7a"
server_key_iv: "ef2626108c82e8fbaa407bc8e18e81ac"
server_key_version: "encryptionkey@0"
I20260925 04:59:56.208824 27465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:59:56.232134 27465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:59:56.236418 27465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:59:56.236719 27465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:59:56.256052 27465 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41679
I20260925 04:59:56.256107 27531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41679 every 8 connection(s)
I20260925 04:59:56.257488 27465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:59:56.261432 27532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:59:56.264660 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 27465
I20260925 04:59:56.264874 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 04:59:56.265319 6262 external_mini_cluster.cc:1428] Setting key 122c7f2713db4c8ac7d6edd7cd60d650
I20260925 04:59:56.274688 27532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 059a762968d54aa381ae04ec3fe086dc: Bootstrap starting.
I20260925 04:59:56.278362 27532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 059a762968d54aa381ae04ec3fe086dc: Neither blocks nor log segments found. Creating new log.
I20260925 04:59:56.279551 27532 log.cc:824] T 00000000000000000000000000000000 P 059a762968d54aa381ae04ec3fe086dc: Log is configured to *not* fsync() on all Append() calls
I20260925 04:59:56.280164 27535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:56.267210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.182:54801 (local address 127.0.0.1:41679)
0925 04:59:56.267978 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:56.267994 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:56.268047 (+ 53us) server_negotiation.cc:407] Connection header received
0925 04:59:56.270192 (+ 2145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:56.270222 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:56.270772 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:56.271221 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:56.271738 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:56.273800 (+ 2062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:56.275319 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:56.276162 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:56.276310 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:56.277633 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:56.277640 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:56.277653 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:59:56.278215 (+ 562us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:56.278293 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:59:56.278319 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:56.278434 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:56.278782 (+ 348us) server_negotiation.cc:299] Negotiation successful
0925 04:59:56.278987 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":298,"threads_started":1}
I20260925 04:59:56.282644 27532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 059a762968d54aa381ae04ec3fe086dc: No bootstrap required, opened a new log
I20260925 04:59:56.284260 6262 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:59:56.290766 27532 raft_consensus.cc:348] T 00000000000000000000000000000000 P 059a762968d54aa381ae04ec3fe086dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "059a762968d54aa381ae04ec3fe086dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41679 } }
I20260925 04:59:56.291244 27532 raft_consensus.cc:374] T 00000000000000000000000000000000 P 059a762968d54aa381ae04ec3fe086dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:59:56.291364 27532 raft_consensus.cc:729] T 00000000000000000000000000000000 P 059a762968d54aa381ae04ec3fe086dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 059a762968d54aa381ae04ec3fe086dc, State: Initialized, Role: FOLLOWER
I20260925 04:59:56.292096 27532 consensus_queue.cc:260] T 00000000000000000000000000000000 P 059a762968d54aa381ae04ec3fe086dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "059a762968d54aa381ae04ec3fe086dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41679 } }
I20260925 04:59:56.292411 27532 raft_consensus.cc:388] T 00000000000000000000000000000000 P 059a762968d54aa381ae04ec3fe086dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:59:56.292554 27532 raft_consensus.cc:482] T 00000000000000000000000000000000 P 059a762968d54aa381ae04ec3fe086dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:59:56.292764 27532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 059a762968d54aa381ae04ec3fe086dc [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:59:56.295563 27532 raft_consensus.cc:504] T 00000000000000000000000000000000 P 059a762968d54aa381ae04ec3fe086dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "059a762968d54aa381ae04ec3fe086dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41679 } }
I20260925 04:59:56.296118 27532 leader_election.cc:302] T 00000000000000000000000000000000 P 059a762968d54aa381ae04ec3fe086dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 059a762968d54aa381ae04ec3fe086dc; no voters:
I20260925 04:59:56.296267 27535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:56.288659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.182:51567 (local address 127.0.0.1:41679)
0925 04:59:56.288850 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:56.288856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:56.289359 (+ 503us) server_negotiation.cc:407] Connection header received
0925 04:59:56.289512 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:56.289517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:56.289597 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:56.289711 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:56.290149 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:56.291245 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:56.293148 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 04:59:56.293900 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 04:59:56.294057 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 04:59:56.295363 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:56.295369 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:56.295374 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:59:56.295403 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:56.295459 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:59:56.295476 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:56.295616 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:56.295867 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 04:59:56.295926 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 04:59:56.296859 27532 leader_election.cc:288] T 00000000000000000000000000000000 P 059a762968d54aa381ae04ec3fe086dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:59:56.296986 27542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 059a762968d54aa381ae04ec3fe086dc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:59:56.298288 27542 raft_consensus.cc:686] T 00000000000000000000000000000000 P 059a762968d54aa381ae04ec3fe086dc [term 1 LEADER]: Becoming Leader. State: Replica: 059a762968d54aa381ae04ec3fe086dc, State: Running, Role: LEADER
I20260925 04:59:56.298882 27542 consensus_queue.cc:237] T 00000000000000000000000000000000 P 059a762968d54aa381ae04ec3fe086dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "059a762968d54aa381ae04ec3fe086dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41679 } }
I20260925 04:59:56.299348 27532 sys_catalog.cc:564] T 00000000000000000000000000000000 P 059a762968d54aa381ae04ec3fe086dc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:59:56.301908 27544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 059a762968d54aa381ae04ec3fe086dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 059a762968d54aa381ae04ec3fe086dc. Latest consensus state: current_term: 1 leader_uuid: "059a762968d54aa381ae04ec3fe086dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "059a762968d54aa381ae04ec3fe086dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41679 } } }
I20260925 04:59:56.302377 27544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 059a762968d54aa381ae04ec3fe086dc [sys.catalog]: This master's current role is: LEADER
I20260925 04:59:56.303059 27543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 059a762968d54aa381ae04ec3fe086dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "059a762968d54aa381ae04ec3fe086dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "059a762968d54aa381ae04ec3fe086dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41679 } } }
I20260925 04:59:56.303459 27543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 059a762968d54aa381ae04ec3fe086dc [sys.catalog]: This master's current role is: LEADER
I20260925 04:59:56.305975 27550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:59:56.312112 27550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:59:56.321122 27550 catalog_manager.cc:1347] Generated new cluster ID: ab66702ae6ca4d2d9637565baca44de7
I20260925 04:59:56.321256 27550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:59:56.339313 27550 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:59:56.340782 27550 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:59:56.352768 27550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 059a762968d54aa381ae04ec3fe086dc: Generated new TSK 0
I20260925 04:59:56.353945 27550 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:59:56.367681 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 27465
W20260925 04:59:56.368980 27527 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.182:54801
W20260925 04:59:56.369791 6262 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 27465: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 04:59:56.369944 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 27465
W20260925 04:59:56.386962 27450 connection.cc:535] client connection to 127.0.0.1:41679 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41679): Connection reset by peer (error 104)
2026-09-25T04:59:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1913 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:59:56 dist-test-slave-xzjr krb5kdc[27564](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:59:56 dist-test-slave-xzjr krb5kdc[27564](info): set up 2 sockets
Sep 25 04:59:56 dist-test-slave-xzjr krb5kdc[27564](info): commencing operation
krb5kdc: starting...
W20260925 04:59:57.472846 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 04:59:57 dist-test-slave-xzjr krb5kdc[27564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:59:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:59:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:59:57.740335 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45681
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 04:59:58.073457 27577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 04:59:58.073752 27577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 04:59:58.073920 27577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 04:59:58.081278 27577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 04:59:58.081399 27577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 04:59:58.081527 27577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 04:59:58.081611 27577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 04:59:58.081688 27577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 04:59:58.091784 27577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45681
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 04:59:58.093238 27577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 04:59:58.094602 27577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 04:59:58.107064 27585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:58.107005 27582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 04:59:58.107070 27583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 04:59:58.107717 27577 server_base.cc:1034] running on GCE node
I20260925 04:59:58.108577 27577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 04:59:58.110056 27577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 04:59:58.111285 27577 hybrid_clock.cc:648] HybridClock initialized: now 1790312398111242 us; error 35 us; skew 500 ppm
I20260925 04:59:58.111605 27577 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 04:59:58 dist-test-slave-xzjr krb5kdc[27564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312398, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 04:59:58.121176 27577 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 04:59:58.123006 27577 webserver.cc:460] Webserver started at http://127.0.0.1:44639/ using document root <none> and password file <none>
I20260925 04:59:58.123951 27577 fs_manager.cc:359] Metadata directory not provided
I20260925 04:59:58.124104 27577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 04:59:58.124467 27577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 04:59:58.127638 27577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "607bc55c9e2c4d5c969bf60806a454d1"
format_stamp: "Formatted at 2026-09-25 04:59:58 on dist-test-slave-xzjr"
server_key: "fddbc77b7dafc28b2384880a7114c93e"
server_key_iv: "f5baaf4fab5ef14211627874b4d520e2"
server_key_version: "encryptionkey@0"
I20260925 04:59:58.128440 27577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "607bc55c9e2c4d5c969bf60806a454d1"
format_stamp: "Formatted at 2026-09-25 04:59:58 on dist-test-slave-xzjr"
server_key: "fddbc77b7dafc28b2384880a7114c93e"
server_key_iv: "f5baaf4fab5ef14211627874b4d520e2"
server_key_version: "encryptionkey@0"
I20260925 04:59:58.133625 27577 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 04:59:58.137162 27592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 04:59:58.138556 27577 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 04:59:58.138723 27577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "607bc55c9e2c4d5c969bf60806a454d1"
format_stamp: "Formatted at 2026-09-25 04:59:58 on dist-test-slave-xzjr"
server_key: "fddbc77b7dafc28b2384880a7114c93e"
server_key_iv: "f5baaf4fab5ef14211627874b4d520e2"
server_key_version: "encryptionkey@0"
I20260925 04:59:58.138906 27577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 04:59:58.144745 27577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 04:59:58.149156 27577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 04:59:58.149425 27577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 04:59:58.168856 27577 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45681
I20260925 04:59:58.168901 27643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45681 every 8 connection(s)
I20260925 04:59:58.170346 27577 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 04:59:58.174125 27644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 04:59:58.176229 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 27577
I20260925 04:59:58.176405 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 04:59:58.176982 6262 external_mini_cluster.cc:1428] Setting key d7f1ed515785e8a109aea2205b3ee314
I20260925 04:59:58.184795 27647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:58.178848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.182:52493 (local address 127.0.0.1:45681)
0925 04:59:58.179668 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:58.179680 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:58.179724 (+ 44us) server_negotiation.cc:407] Connection header received
0925 04:59:58.181357 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:58.181385 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:58.181750 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:58.182077 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:58.183144 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:58.183151 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:58.183164 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:59:58.183180 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:58.183240 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:59:58.183259 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:58.183430 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:58.183561 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 04:59:58.183841 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":352,"threads_started":1}
I20260925 04:59:58.188514 27644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 607bc55c9e2c4d5c969bf60806a454d1: Bootstrap starting.
I20260925 04:59:58.188751 6262 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 04:59:58.191576 27644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 607bc55c9e2c4d5c969bf60806a454d1: Neither blocks nor log segments found. Creating new log.
I20260925 04:59:58.192662 27644 log.cc:824] T 00000000000000000000000000000000 P 607bc55c9e2c4d5c969bf60806a454d1: Log is configured to *not* fsync() on all Append() calls
I20260925 04:59:58.195677 27647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 04:59:58.193116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.182:39543 (local address 127.0.0.1:45681)
0925 04:59:58.193326 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 04:59:58.193334 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 04:59:58.193650 (+ 316us) server_negotiation.cc:407] Connection header received
0925 04:59:58.193785 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 04:59:58.193794 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 04:59:58.193870 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 04:59:58.193989 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 04:59:58.194992 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 04:59:58.194997 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 04:59:58.195001 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 04:59:58.195018 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 04:59:58.195063 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 04:59:58.195083 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 04:59:58.195209 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 04:59:58.195336 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 04:59:58.195392 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 04:59:58.196691 27644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 607bc55c9e2c4d5c969bf60806a454d1: No bootstrap required, opened a new log
I20260925 04:59:58.203944 27644 raft_consensus.cc:348] T 00000000000000000000000000000000 P 607bc55c9e2c4d5c969bf60806a454d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "607bc55c9e2c4d5c969bf60806a454d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45681 } }
I20260925 04:59:58.204272 27644 raft_consensus.cc:374] T 00000000000000000000000000000000 P 607bc55c9e2c4d5c969bf60806a454d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 04:59:58.204344 27644 raft_consensus.cc:729] T 00000000000000000000000000000000 P 607bc55c9e2c4d5c969bf60806a454d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 607bc55c9e2c4d5c969bf60806a454d1, State: Initialized, Role: FOLLOWER
I20260925 04:59:58.204895 27644 consensus_queue.cc:260] T 00000000000000000000000000000000 P 607bc55c9e2c4d5c969bf60806a454d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "607bc55c9e2c4d5c969bf60806a454d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45681 } }
I20260925 04:59:58.205137 27644 raft_consensus.cc:388] T 00000000000000000000000000000000 P 607bc55c9e2c4d5c969bf60806a454d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 04:59:58.205289 27644 raft_consensus.cc:482] T 00000000000000000000000000000000 P 607bc55c9e2c4d5c969bf60806a454d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 04:59:58.205458 27644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 607bc55c9e2c4d5c969bf60806a454d1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 04:59:58.207820 27644 raft_consensus.cc:504] T 00000000000000000000000000000000 P 607bc55c9e2c4d5c969bf60806a454d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "607bc55c9e2c4d5c969bf60806a454d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45681 } }
I20260925 04:59:58.208312 27644 leader_election.cc:302] T 00000000000000000000000000000000 P 607bc55c9e2c4d5c969bf60806a454d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 607bc55c9e2c4d5c969bf60806a454d1; no voters:
I20260925 04:59:58.208871 27644 leader_election.cc:288] T 00000000000000000000000000000000 P 607bc55c9e2c4d5c969bf60806a454d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 04:59:58.209009 27654 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 607bc55c9e2c4d5c969bf60806a454d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 04:59:58.209686 27654 raft_consensus.cc:686] T 00000000000000000000000000000000 P 607bc55c9e2c4d5c969bf60806a454d1 [term 1 LEADER]: Becoming Leader. State: Replica: 607bc55c9e2c4d5c969bf60806a454d1, State: Running, Role: LEADER
I20260925 04:59:58.210239 27654 consensus_queue.cc:237] T 00000000000000000000000000000000 P 607bc55c9e2c4d5c969bf60806a454d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "607bc55c9e2c4d5c969bf60806a454d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45681 } }
I20260925 04:59:58.210947 27644 sys_catalog.cc:564] T 00000000000000000000000000000000 P 607bc55c9e2c4d5c969bf60806a454d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 04:59:58.213008 27656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 607bc55c9e2c4d5c969bf60806a454d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 607bc55c9e2c4d5c969bf60806a454d1. Latest consensus state: current_term: 1 leader_uuid: "607bc55c9e2c4d5c969bf60806a454d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "607bc55c9e2c4d5c969bf60806a454d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45681 } } }
I20260925 04:59:58.213402 27656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 607bc55c9e2c4d5c969bf60806a454d1 [sys.catalog]: This master's current role is: LEADER
I20260925 04:59:58.218015 27655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 607bc55c9e2c4d5c969bf60806a454d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "607bc55c9e2c4d5c969bf60806a454d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "607bc55c9e2c4d5c969bf60806a454d1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45681 } } }
I20260925 04:59:58.218386 27655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 607bc55c9e2c4d5c969bf60806a454d1 [sys.catalog]: This master's current role is: LEADER
I20260925 04:59:58.228086 27661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 04:59:58.233572 27661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 04:59:58.243589 27661 catalog_manager.cc:1347] Generated new cluster ID: 0532d4a6126b4196b738e4f7b53d277f
I20260925 04:59:58.243733 27661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 04:59:58.266153 27661 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 04:59:58.266695 27661 catalog_manager.cc:1504] Loading token signing keys...
I20260925 04:59:58.272677 27661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 607bc55c9e2c4d5c969bf60806a454d1: Generated new TSK 0
I20260925 04:59:58.273500 27661 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 04:59:58.282652 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 27577
W20260925 04:59:58.284238 27639 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.182:52493
W20260925 04:59:58.284934 6262 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 27577: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 04:59:58.285091 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 27577
2026-09-25T04:59:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1912 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 04:59:58 dist-test-slave-xzjr krb5kdc[27675](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 04:59:58 dist-test-slave-xzjr krb5kdc[27675](info): set up 2 sockets
Sep 25 04:59:58 dist-test-slave-xzjr krb5kdc[27675](info): commencing operation
krb5kdc: starting...
W20260925 04:59:59.392622 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 04:59:59 dist-test-slave-xzjr krb5kdc[27675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T04:59:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T04:59:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 04:59:59.659348 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39447
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:00:00.013958 27688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:00:00.014313 27688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:00:00.014483 27688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:00:00.021920 27688 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:00:00.022080 27688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:00:00.022186 27688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:00:00.022290 27688 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:00:00.022363 27688 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:00:00.032846 27688 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39447
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:00:00.034322 27688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:00:00.035661 27688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:00:00.047798 27696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:00.047865 27694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:00.047861 27693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:00:00.048719 27688 server_base.cc:1034] running on GCE node
I20260925 05:00:00.049460 27688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:00:00.050839 27688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:00:00.052073 27688 hybrid_clock.cc:648] HybridClock initialized: now 1790312400052028 us; error 79 us; skew 500 ppm
I20260925 05:00:00.052486 27688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:00:00 dist-test-slave-xzjr krb5kdc[27675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:00:00.055552 27688 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:00:00.057096 27688 webserver.cc:460] Webserver started at http://127.0.0.1:46129/ using document root <none> and password file <none>
I20260925 05:00:00.057801 27688 fs_manager.cc:359] Metadata directory not provided
I20260925 05:00:00.057943 27688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:00:00.058295 27688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:00:00.061345 27688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "6c1ac3d9fe314647b6d4a5c2c5f21d1e"
format_stamp: "Formatted at 2026-09-25 05:00:00 on dist-test-slave-xzjr"
server_key: "572abc7c9c023983658df50c45009c99"
server_key_iv: "b0175e3962b59addfd6a179ec98452a0"
server_key_version: "encryptionkey@0"
I20260925 05:00:00.062136 27688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "6c1ac3d9fe314647b6d4a5c2c5f21d1e"
format_stamp: "Formatted at 2026-09-25 05:00:00 on dist-test-slave-xzjr"
server_key: "572abc7c9c023983658df50c45009c99"
server_key_iv: "b0175e3962b59addfd6a179ec98452a0"
server_key_version: "encryptionkey@0"
I20260925 05:00:00.067400 27688 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260925 05:00:00.070919 27703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:00.072314 27688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:00:00.072515 27688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "6c1ac3d9fe314647b6d4a5c2c5f21d1e"
format_stamp: "Formatted at 2026-09-25 05:00:00 on dist-test-slave-xzjr"
server_key: "572abc7c9c023983658df50c45009c99"
server_key_iv: "b0175e3962b59addfd6a179ec98452a0"
server_key_version: "encryptionkey@0"
I20260925 05:00:00.072726 27688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:00:00.113503 27688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:00:00.118237 27688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:00:00.118548 27688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:00:00.137578 27688 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39447
I20260925 05:00:00.137609 27754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39447 every 8 connection(s)
I20260925 05:00:00.138984 27688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 05:00:00.143042 27755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:00.145709 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 27688
I20260925 05:00:00.145903 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 05:00:00.146322 6262 external_mini_cluster.cc:1428] Setting key 7d009656b62813a94fa7df266f2ab6b3
I20260925 05:00:00.157238 27755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c1ac3d9fe314647b6d4a5c2c5f21d1e: Bootstrap starting.
I20260925 05:00:00.161024 27755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c1ac3d9fe314647b6d4a5c2c5f21d1e: Neither blocks nor log segments found. Creating new log.
Sep 25 05:00:00 dist-test-slave-xzjr krb5kdc[27675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 05:00:00.162297 27755 log.cc:824] T 00000000000000000000000000000000 P 6c1ac3d9fe314647b6d4a5c2c5f21d1e: Log is configured to *not* fsync() on all Append() calls
I20260925 05:00:00.166159 27755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c1ac3d9fe314647b6d4a5c2c5f21d1e: No bootstrap required, opened a new log
I20260925 05:00:00.172721 27758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:00.148302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.182:47513 (local address 127.0.0.1:39447)
0925 05:00:00.149198 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:00.149220 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:00.149278 (+ 58us) server_negotiation.cc:407] Connection header received
0925 05:00:00.151740 (+ 2462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:00.151797 (+ 57us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:00.152360 (+ 563us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:00.152873 (+ 513us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:00.154195 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:00.156322 (+ 2127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:00.157697 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:00.158738 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:00.158939 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:00.163568 (+ 4629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:00.163577 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:00.163593 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:00.163631 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:00.167705 (+ 4074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:00.168584 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:00.168593 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:00.168602 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:00.168832 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:00.169271 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:00.169280 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:00.169284 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:00.170049 (+ 765us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:00.170380 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:00.170837 (+ 457us) server_negotiation.cc:299] Negotiation successful
0925 05:00:00.171201 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":401,"threads_started":1}
I20260925 05:00:00.176345 27755 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c1ac3d9fe314647b6d4a5c2c5f21d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c1ac3d9fe314647b6d4a5c2c5f21d1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39447 } }
I20260925 05:00:00.176743 27755 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c1ac3d9fe314647b6d4a5c2c5f21d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:00:00.176899 27755 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c1ac3d9fe314647b6d4a5c2c5f21d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c1ac3d9fe314647b6d4a5c2c5f21d1e, State: Initialized, Role: FOLLOWER
I20260925 05:00:00.176932 6262 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:00:00.177491 27755 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c1ac3d9fe314647b6d4a5c2c5f21d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c1ac3d9fe314647b6d4a5c2c5f21d1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39447 } }
I20260925 05:00:00.177745 27755 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c1ac3d9fe314647b6d4a5c2c5f21d1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:00:00.177887 27755 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c1ac3d9fe314647b6d4a5c2c5f21d1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:00:00.178078 27755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c1ac3d9fe314647b6d4a5c2c5f21d1e [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:00.180552 27755 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c1ac3d9fe314647b6d4a5c2c5f21d1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c1ac3d9fe314647b6d4a5c2c5f21d1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39447 } }
I20260925 05:00:00.181087 27755 leader_election.cc:302] T 00000000000000000000000000000000 P 6c1ac3d9fe314647b6d4a5c2c5f21d1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c1ac3d9fe314647b6d4a5c2c5f21d1e; no voters:
I20260925 05:00:00.182024 27755 leader_election.cc:288] T 00000000000000000000000000000000 P 6c1ac3d9fe314647b6d4a5c2c5f21d1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:00:00.182188 27764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c1ac3d9fe314647b6d4a5c2c5f21d1e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:00:00.184103 27764 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c1ac3d9fe314647b6d4a5c2c5f21d1e [term 1 LEADER]: Becoming Leader. State: Replica: 6c1ac3d9fe314647b6d4a5c2c5f21d1e, State: Running, Role: LEADER
I20260925 05:00:00.184710 27764 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c1ac3d9fe314647b6d4a5c2c5f21d1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c1ac3d9fe314647b6d4a5c2c5f21d1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39447 } }
I20260925 05:00:00.185436 27755 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c1ac3d9fe314647b6d4a5c2c5f21d1e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:00:00.188947 27766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c1ac3d9fe314647b6d4a5c2c5f21d1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c1ac3d9fe314647b6d4a5c2c5f21d1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c1ac3d9fe314647b6d4a5c2c5f21d1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39447 } } }
I20260925 05:00:00.189365 27766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c1ac3d9fe314647b6d4a5c2c5f21d1e [sys.catalog]: This master's current role is: LEADER
I20260925 05:00:00.189596 27767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c1ac3d9fe314647b6d4a5c2c5f21d1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c1ac3d9fe314647b6d4a5c2c5f21d1e. Latest consensus state: current_term: 1 leader_uuid: "6c1ac3d9fe314647b6d4a5c2c5f21d1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c1ac3d9fe314647b6d4a5c2c5f21d1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39447 } } }
I20260925 05:00:00.193639 27767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c1ac3d9fe314647b6d4a5c2c5f21d1e [sys.catalog]: This master's current role is: LEADER
I20260925 05:00:00.199736 27770 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:00:00.201508 27758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:00.181671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.182:46195 (local address 127.0.0.1:39447)
0925 05:00:00.181905 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:00.181912 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:00.183188 (+ 1276us) server_negotiation.cc:407] Connection header received
0925 05:00:00.183394 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:00.183401 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:00.183503 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:00.183610 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:00.185527 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:00.186387 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:00.187788 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:00.188546 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:00.188699 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:00.192839 (+ 4140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:00.192849 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:00.192857 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:00.192912 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:00.197817 (+ 4905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:00.199129 (+ 1312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:00.199138 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:00.199145 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:00.199322 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:00.199807 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:00.199813 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:00.199818 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:00.200289 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:00.200560 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:00.200938 (+ 378us) server_negotiation.cc:299] Negotiation successful
0925 05:00:00.201127 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 05:00:00.206830 27770 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:00:00.221086 27770 catalog_manager.cc:1347] Generated new cluster ID: f5e47bea29774a389cae7a26f94411cb
I20260925 05:00:00.221243 27770 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:00:00.275219 27770 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:00:00.276783 27770 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:00:00.296394 27770 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c1ac3d9fe314647b6d4a5c2c5f21d1e: Generated new TSK 0
I20260925 05:00:00.297310 27770 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:00:00.314281 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 27688
I20260925 05:00:00.315536 27750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:00:00.414053 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 27688
W20260925 05:00:00.431680 27673 connection.cc:535] client connection to 127.0.0.1:39447 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39447): Connection reset by peer (error 104)
2026-09-25T05:00:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2131 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:00:00 dist-test-slave-xzjr krb5kdc[27787](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:00:00 dist-test-slave-xzjr krb5kdc[27787](info): set up 2 sockets
Sep 25 05:00:00 dist-test-slave-xzjr krb5kdc[27787](info): commencing operation
krb5kdc: starting...
W20260925 05:00:01.524782 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 05:00:01 dist-test-slave-xzjr krb5kdc[27787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:00:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:00:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:00:01.797318 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40699
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:00:02.138252 27800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:00:02.138558 27800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:00:02.138729 27800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:00:02.146255 27800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:00:02.146375 27800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:00:02.146492 27800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:00:02.146600 27800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:00:02.146670 27800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:00:02.156863 27800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40699
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:00:02.158296 27800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:00:02.159674 27800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:00:02.172365 27808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:02.172593 27806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:02.172621 27805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:00:02.174292 27800 server_base.cc:1034] running on GCE node
I20260925 05:00:02.175148 27800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:00:02.176739 27800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:00:02.177996 27800 hybrid_clock.cc:648] HybridClock initialized: now 1790312402177949 us; error 63 us; skew 500 ppm
I20260925 05:00:02.178386 27800 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:00:02 dist-test-slave-xzjr krb5kdc[27787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:00:02.181763 27800 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:00:02.183436 27800 webserver.cc:460] Webserver started at http://127.0.0.1:43227/ using document root <none> and password file <none>
I20260925 05:00:02.184233 27800 fs_manager.cc:359] Metadata directory not provided
I20260925 05:00:02.184384 27800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:00:02.184707 27800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:00:02.187942 27800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "dee65a47155d4106a500037332f01c5d"
format_stamp: "Formatted at 2026-09-25 05:00:02 on dist-test-slave-xzjr"
server_key: "d455cb8e24d2aba22514740d495701a9"
server_key_iv: "bd40f5f87cc944bce3ef2c91613998af"
server_key_version: "encryptionkey@0"
I20260925 05:00:02.188752 27800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "dee65a47155d4106a500037332f01c5d"
format_stamp: "Formatted at 2026-09-25 05:00:02 on dist-test-slave-xzjr"
server_key: "d455cb8e24d2aba22514740d495701a9"
server_key_iv: "bd40f5f87cc944bce3ef2c91613998af"
server_key_version: "encryptionkey@0"
I20260925 05:00:02.194228 27800 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 05:00:02.197897 27815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:02.199311 27800 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:00:02.199497 27800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "dee65a47155d4106a500037332f01c5d"
format_stamp: "Formatted at 2026-09-25 05:00:02 on dist-test-slave-xzjr"
server_key: "d455cb8e24d2aba22514740d495701a9"
server_key_iv: "bd40f5f87cc944bce3ef2c91613998af"
server_key_version: "encryptionkey@0"
I20260925 05:00:02.199709 27800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:00:02.235105 27800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:00:02.239666 27800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:00:02.240015 27800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:00:02.259532 27800 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40699
I20260925 05:00:02.259555 27866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40699 every 8 connection(s)
I20260925 05:00:02.261021 27800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 05:00:02.264191 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 27800
I20260925 05:00:02.264376 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 05:00:02.264911 6262 external_mini_cluster.cc:1428] Setting key fe7fe1a40ef881880f3e5e27637d2b83
I20260925 05:00:02.265424 27867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:02.280009 27867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dee65a47155d4106a500037332f01c5d: Bootstrap starting.
I20260925 05:00:02.280066 27870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:02.266809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.182:49115 (local address 127.0.0.1:40699)
0925 05:00:02.267635 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:02.267647 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:02.267689 (+ 42us) server_negotiation.cc:407] Connection header received
0925 05:00:02.270251 (+ 2562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:02.270275 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:02.270726 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:02.271085 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:02.271740 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:02.273190 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:02.275090 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:02.275903 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:02.276054 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:02.277488 (+ 1434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:02.277496 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:02.277510 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:00:02.278128 (+ 618us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:02.278213 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:00:02.278239 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:02.278347 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:02.278735 (+ 388us) server_negotiation.cc:299] Negotiation successful
0925 05:00:02.278942 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":345,"threads_started":1}
I20260925 05:00:02.283735 27867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dee65a47155d4106a500037332f01c5d: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:02.284282 6262 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:00:02.284991 27867 log.cc:824] T 00000000000000000000000000000000 P dee65a47155d4106a500037332f01c5d: Log is configured to *not* fsync() on all Append() calls
I20260925 05:00:02.287853 27867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dee65a47155d4106a500037332f01c5d: No bootstrap required, opened a new log
I20260925 05:00:02.295588 27870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:02.288787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.182:46771 (local address 127.0.0.1:40699)
0925 05:00:02.288976 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:02.288982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:02.289391 (+ 409us) server_negotiation.cc:407] Connection header received
0925 05:00:02.289540 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:02.289546 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:02.289630 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:02.289739 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:02.290173 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:02.291372 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:02.292745 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:02.293496 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:02.293630 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:02.294777 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:02.294783 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:02.294788 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:00:02.294807 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:02.294852 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:00:02.294871 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:02.294972 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:02.295151 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 05:00:02.295213 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 05:00:02.297456 27867 raft_consensus.cc:348] T 00000000000000000000000000000000 P dee65a47155d4106a500037332f01c5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee65a47155d4106a500037332f01c5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40699 } }
I20260925 05:00:02.297812 27867 raft_consensus.cc:374] T 00000000000000000000000000000000 P dee65a47155d4106a500037332f01c5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:00:02.297890 27867 raft_consensus.cc:729] T 00000000000000000000000000000000 P dee65a47155d4106a500037332f01c5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dee65a47155d4106a500037332f01c5d, State: Initialized, Role: FOLLOWER
I20260925 05:00:02.298488 27867 consensus_queue.cc:260] T 00000000000000000000000000000000 P dee65a47155d4106a500037332f01c5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee65a47155d4106a500037332f01c5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40699 } }
I20260925 05:00:02.298730 27867 raft_consensus.cc:388] T 00000000000000000000000000000000 P dee65a47155d4106a500037332f01c5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:00:02.298895 27867 raft_consensus.cc:482] T 00000000000000000000000000000000 P dee65a47155d4106a500037332f01c5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:00:02.299074 27867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dee65a47155d4106a500037332f01c5d [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:02.301366 27867 raft_consensus.cc:504] T 00000000000000000000000000000000 P dee65a47155d4106a500037332f01c5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee65a47155d4106a500037332f01c5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40699 } }
I20260925 05:00:02.301883 27867 leader_election.cc:302] T 00000000000000000000000000000000 P dee65a47155d4106a500037332f01c5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dee65a47155d4106a500037332f01c5d; no voters:
I20260925 05:00:02.302531 27867 leader_election.cc:288] T 00000000000000000000000000000000 P dee65a47155d4106a500037332f01c5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:00:02.302699 27877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dee65a47155d4106a500037332f01c5d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:00:02.303933 27877 raft_consensus.cc:686] T 00000000000000000000000000000000 P dee65a47155d4106a500037332f01c5d [term 1 LEADER]: Becoming Leader. State: Replica: dee65a47155d4106a500037332f01c5d, State: Running, Role: LEADER
I20260925 05:00:02.304649 27877 consensus_queue.cc:237] T 00000000000000000000000000000000 P dee65a47155d4106a500037332f01c5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee65a47155d4106a500037332f01c5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40699 } }
I20260925 05:00:02.305630 27867 sys_catalog.cc:564] T 00000000000000000000000000000000 P dee65a47155d4106a500037332f01c5d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:00:02.307821 27879 sys_catalog.cc:455] T 00000000000000000000000000000000 P dee65a47155d4106a500037332f01c5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader dee65a47155d4106a500037332f01c5d. Latest consensus state: current_term: 1 leader_uuid: "dee65a47155d4106a500037332f01c5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee65a47155d4106a500037332f01c5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40699 } } }
I20260925 05:00:02.307807 27878 sys_catalog.cc:455] T 00000000000000000000000000000000 P dee65a47155d4106a500037332f01c5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dee65a47155d4106a500037332f01c5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dee65a47155d4106a500037332f01c5d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40699 } } }
I20260925 05:00:02.308359 27879 sys_catalog.cc:458] T 00000000000000000000000000000000 P dee65a47155d4106a500037332f01c5d [sys.catalog]: This master's current role is: LEADER
I20260925 05:00:02.308477 27878 sys_catalog.cc:458] T 00000000000000000000000000000000 P dee65a47155d4106a500037332f01c5d [sys.catalog]: This master's current role is: LEADER
I20260925 05:00:02.310016 27886 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:00:02.315223 27886 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:00:02.328404 27886 catalog_manager.cc:1347] Generated new cluster ID: 92fd11951fee4501853c8dcad6a596c9
I20260925 05:00:02.328537 27886 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:00:02.342191 27886 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:00:02.343631 27886 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:00:02.351192 27886 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dee65a47155d4106a500037332f01c5d: Generated new TSK 0
I20260925 05:00:02.352171 27886 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:00:02.362233 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 27800
W20260925 05:00:02.363406 27862 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.182:49115
W20260925 05:00:02.364252 6262 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 27800: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:00:02.364411 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 27800
W20260925 05:00:02.383996 27785 connection.cc:535] client connection to 127.0.0.1:40699 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40699): Connection reset by peer (error 104)
2026-09-25T05:00:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1955 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:00:02 dist-test-slave-xzjr krb5kdc[27898](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:00:02 dist-test-slave-xzjr krb5kdc[27898](info): set up 2 sockets
Sep 25 05:00:02 dist-test-slave-xzjr krb5kdc[27898](info): commencing operation
krb5kdc: starting...
W20260925 05:00:03.512686 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 05:00:03 dist-test-slave-xzjr krb5kdc[27898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:00:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:00:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 05:00:03.774609 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46795
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:00:04.101583 27911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:00:04.101933 27911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:00:04.102126 27911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:00:04.109427 27911 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:00:04.109586 27911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:00:04.109694 27911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:00:04.109789 27911 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:00:04.109856 27911 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:00:04.119866 27911 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46795
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312053666503-6262-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:00:04.121227 27911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:00:04.122558 27911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:00:04.134624 27917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:04.134650 27916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:04.135102 27919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:00:04.135398 27911 server_base.cc:1034] running on GCE node
I20260925 05:00:04.136270 27911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:00:04.137737 27911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:00:04.138945 27911 hybrid_clock.cc:648] HybridClock initialized: now 1790312404138902 us; error 30 us; skew 500 ppm
I20260925 05:00:04.139357 27911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:00:04 dist-test-slave-xzjr krb5kdc[27898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:00:04.147287 27911 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 05:00:04.148903 27911 webserver.cc:460] Webserver started at http://127.0.0.1:43413/ using document root <none> and password file <none>
I20260925 05:00:04.149598 27911 fs_manager.cc:359] Metadata directory not provided
I20260925 05:00:04.149750 27911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:00:04.150060 27911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:00:04.153095 27911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "fc7429c5ce6948bda682257fbb18b89a"
format_stamp: "Formatted at 2026-09-25 05:00:04 on dist-test-slave-xzjr"
server_key: "a50885e3117e3296cc9404400394d605"
server_key_iv: "676bb75d540a16da4986baba0e4f7d7b"
server_key_version: "encryptionkey@0"
I20260925 05:00:04.153890 27911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "fc7429c5ce6948bda682257fbb18b89a"
format_stamp: "Formatted at 2026-09-25 05:00:04 on dist-test-slave-xzjr"
server_key: "a50885e3117e3296cc9404400394d605"
server_key_iv: "676bb75d540a16da4986baba0e4f7d7b"
server_key_version: "encryptionkey@0"
I20260925 05:00:04.159183 27911 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 05:00:04.162771 27926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:04.164166 27911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:00:04.164363 27911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "fc7429c5ce6948bda682257fbb18b89a"
format_stamp: "Formatted at 2026-09-25 05:00:04 on dist-test-slave-xzjr"
server_key: "a50885e3117e3296cc9404400394d605"
server_key_iv: "676bb75d540a16da4986baba0e4f7d7b"
server_key_version: "encryptionkey@0"
I20260925 05:00:04.164568 27911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:00:04.170331 27911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:00:04.174244 27911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:00:04.174532 27911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:00:04.193084 27911 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46795
I20260925 05:00:04.193133 27977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46795 every 8 connection(s)
I20260925 05:00:04.194511 27911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 05:00:04.198282 27978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:04.198559 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 27911
I20260925 05:00:04.198732 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 05:00:04.199182 6262 external_mini_cluster.cc:1428] Setting key 8f22afc93b5418bce6be2e6a29befc2f
I20260925 05:00:04.207427 27981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:04.201077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.182:51975 (local address 127.0.0.1:46795)
0925 05:00:04.201878 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:04.201890 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:04.201932 (+ 42us) server_negotiation.cc:407] Connection header received
0925 05:00:04.204178 (+ 2246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:04.204205 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:04.204546 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:04.204875 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:04.205905 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:04.205911 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:04.205924 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:00:04.205938 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:04.205995 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:00:04.206025 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:04.206184 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:04.206319 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 05:00:04.206543 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":357,"threads_started":1}
I20260925 05:00:04.210880 27978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc7429c5ce6948bda682257fbb18b89a: Bootstrap starting.
I20260925 05:00:04.211066 6262 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:00:04.213881 27978 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc7429c5ce6948bda682257fbb18b89a: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:04.215340 27978 log.cc:824] T 00000000000000000000000000000000 P fc7429c5ce6948bda682257fbb18b89a: Log is configured to *not* fsync() on all Append() calls
I20260925 05:00:04.221010 27981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:04.218302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.182:40029 (local address 127.0.0.1:46795)
0925 05:00:04.218559 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:04.218570 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:04.218590 (+ 20us) server_negotiation.cc:407] Connection header received
0925 05:00:04.218705 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:04.218710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:04.218768 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:04.218876 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:04.220095 (+ 1219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:04.220101 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:04.220107 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:00:04.220129 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:04.220173 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:00:04.220195 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:04.220310 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:04.220535 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 05:00:04.220624 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 05:00:04.221369 27978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc7429c5ce6948bda682257fbb18b89a: No bootstrap required, opened a new log
I20260925 05:00:04.229525 27978 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc7429c5ce6948bda682257fbb18b89a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc7429c5ce6948bda682257fbb18b89a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46795 } }
I20260925 05:00:04.229903 27978 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc7429c5ce6948bda682257fbb18b89a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:00:04.230005 27978 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc7429c5ce6948bda682257fbb18b89a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc7429c5ce6948bda682257fbb18b89a, State: Initialized, Role: FOLLOWER
I20260925 05:00:04.230563 27978 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc7429c5ce6948bda682257fbb18b89a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc7429c5ce6948bda682257fbb18b89a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46795 } }
I20260925 05:00:04.230801 27978 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc7429c5ce6948bda682257fbb18b89a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:00:04.230950 27978 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc7429c5ce6948bda682257fbb18b89a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:00:04.231124 27978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc7429c5ce6948bda682257fbb18b89a [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:04.233336 27978 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc7429c5ce6948bda682257fbb18b89a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc7429c5ce6948bda682257fbb18b89a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46795 } }
I20260925 05:00:04.233829 27978 leader_election.cc:302] T 00000000000000000000000000000000 P fc7429c5ce6948bda682257fbb18b89a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc7429c5ce6948bda682257fbb18b89a; no voters:
I20260925 05:00:04.234401 27978 leader_election.cc:288] T 00000000000000000000000000000000 P fc7429c5ce6948bda682257fbb18b89a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:00:04.234565 27988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc7429c5ce6948bda682257fbb18b89a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:00:04.235250 27988 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc7429c5ce6948bda682257fbb18b89a [term 1 LEADER]: Becoming Leader. State: Replica: fc7429c5ce6948bda682257fbb18b89a, State: Running, Role: LEADER
I20260925 05:00:04.235833 27988 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc7429c5ce6948bda682257fbb18b89a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc7429c5ce6948bda682257fbb18b89a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46795 } }
I20260925 05:00:04.236706 27978 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc7429c5ce6948bda682257fbb18b89a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:00:04.238796 27990 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc7429c5ce6948bda682257fbb18b89a [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc7429c5ce6948bda682257fbb18b89a. Latest consensus state: current_term: 1 leader_uuid: "fc7429c5ce6948bda682257fbb18b89a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc7429c5ce6948bda682257fbb18b89a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46795 } } }
I20260925 05:00:04.239257 27990 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc7429c5ce6948bda682257fbb18b89a [sys.catalog]: This master's current role is: LEADER
I20260925 05:00:04.239171 27989 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc7429c5ce6948bda682257fbb18b89a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc7429c5ce6948bda682257fbb18b89a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc7429c5ce6948bda682257fbb18b89a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46795 } } }
I20260925 05:00:04.239473 27989 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc7429c5ce6948bda682257fbb18b89a [sys.catalog]: This master's current role is: LEADER
I20260925 05:00:04.241811 27996 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:00:04.247025 27996 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:00:04.255738 27996 catalog_manager.cc:1347] Generated new cluster ID: 0c8869b366224c9da4c05fe083a6bbdf
I20260925 05:00:04.255877 27996 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:00:04.268231 27996 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:00:04.268738 27996 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:00:04.282572 27996 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc7429c5ce6948bda682257fbb18b89a: Generated new TSK 0
I20260925 05:00:04.283722 27996 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:00:04.301424 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 27911
W20260925 05:00:04.303042 27973 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.182:51975
W20260925 05:00:04.303853 6262 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 27911: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:00:04.304004 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 27911
2026-09-25T05:00:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1933 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:00:04 dist-test-slave-xzjr krb5kdc[28009](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:00:04 dist-test-slave-xzjr krb5kdc[28009](info): set up 2 sockets
Sep 25 05:00:04 dist-test-slave-xzjr krb5kdc[28009](info): commencing operation
krb5kdc: starting...
W20260925 05:00:05.420570 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 05:00:05 dist-test-slave-xzjr krb5kdc[28009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:00:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:00:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.29.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
I20260925 05:00:05.679889 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:33913
--webserver_interface=127.6.29.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:34783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:33913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:00:06.012638 28022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:00:06.012924 28022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:00:06.013018 28022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:00:06.020247 28022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:00:06.020366 28022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:00:06.020413 28022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:00:06.020473 28022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:00:06.020516 28022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:00:06.030421 28022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:34783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:33913
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:33913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:00:06.031697 28022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:00:06.033058 28022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:00:06.045168 28027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:06.045202 28028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:06.045457 28030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:00:06.046450 28022 server_base.cc:1034] running on GCE node
I20260925 05:00:06.047273 28022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:00:06.048895 28022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:00:06.050156 28022 hybrid_clock.cc:648] HybridClock initialized: now 1790312406050110 us; error 42 us; skew 500 ppm
I20260925 05:00:06.050544 28022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:00:06 dist-test-slave-xzjr krb5kdc[28009](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312406, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:00:06.053911 28022 init.cc:377] Logged in from keytab as kudu/127.6.29.190@KRBTEST.COM (short username kudu)
I20260925 05:00:06.055483 28022 webserver.cc:460] Webserver started at http://127.6.29.190:38539/ using document root <none> and password file <none>
I20260925 05:00:06.056249 28022 fs_manager.cc:359] Metadata directory not provided
I20260925 05:00:06.056361 28022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:00:06.056622 28022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:00:06.059696 28022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "bc627d7693e64c8cb58e5cdf140d6ecd"
format_stamp: "Formatted at 2026-09-25 05:00:06 on dist-test-slave-xzjr"
server_key: "dfdfff1e0b10e8c18f6adbea02626035"
server_key_iv: "5d4ad39544a05016640ca4dad2ae466b"
server_key_version: "encryptionkey@0"
I20260925 05:00:06.060505 28022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "bc627d7693e64c8cb58e5cdf140d6ecd"
format_stamp: "Formatted at 2026-09-25 05:00:06 on dist-test-slave-xzjr"
server_key: "dfdfff1e0b10e8c18f6adbea02626035"
server_key_iv: "5d4ad39544a05016640ca4dad2ae466b"
server_key_version: "encryptionkey@0"
I20260925 05:00:06.065845 28022 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 05:00:06.069393 28037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:06.070767 28022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 05:00:06.070909 28022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "bc627d7693e64c8cb58e5cdf140d6ecd"
format_stamp: "Formatted at 2026-09-25 05:00:06 on dist-test-slave-xzjr"
server_key: "dfdfff1e0b10e8c18f6adbea02626035"
server_key_iv: "5d4ad39544a05016640ca4dad2ae466b"
server_key_version: "encryptionkey@0"
I20260925 05:00:06.071069 28022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:00:06.090727 28022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:00:06.095106 28022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:00:06.095356 28022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:00:06.114439 28022 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:33913
I20260925 05:00:06.114461 28088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:33913 every 8 connection(s)
I20260925 05:00:06.115971 28022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 05:00:06.119874 28089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:06.125357 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 28022
I20260925 05:00:06.125540 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 05:00:06.125982 6262 external_mini_cluster.cc:1428] Setting key f5f5d534213ac2eba540f1c028484a1f
I20260925 05:00:06.130726 28089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc627d7693e64c8cb58e5cdf140d6ecd: Bootstrap starting.
I20260925 05:00:06.134464 28089 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc627d7693e64c8cb58e5cdf140d6ecd: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:06.135815 28089 log.cc:824] T 00000000000000000000000000000000 P bc627d7693e64c8cb58e5cdf140d6ecd: Log is configured to *not* fsync() on all Append() calls
I20260925 05:00:06.139587 28089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc627d7693e64c8cb58e5cdf140d6ecd: No bootstrap required, opened a new log
Sep 25 05:00:06 dist-test-slave-xzjr krb5kdc[28009](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 05:00:06.147734 28089 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc627d7693e64c8cb58e5cdf140d6ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc627d7693e64c8cb58e5cdf140d6ecd" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33913 } }
I20260925 05:00:06.148236 28089 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc627d7693e64c8cb58e5cdf140d6ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:00:06.148352 28089 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc627d7693e64c8cb58e5cdf140d6ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc627d7693e64c8cb58e5cdf140d6ecd, State: Initialized, Role: FOLLOWER
I20260925 05:00:06.149128 28089 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc627d7693e64c8cb58e5cdf140d6ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc627d7693e64c8cb58e5cdf140d6ecd" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33913 } }
I20260925 05:00:06.149422 28089 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc627d7693e64c8cb58e5cdf140d6ecd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:00:06.149573 28089 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc627d7693e64c8cb58e5cdf140d6ecd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:00:06.149748 28089 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc627d7693e64c8cb58e5cdf140d6ecd [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:06.150974 28092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:06.127815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58783 (local address 127.6.29.190:33913)
0925 05:00:06.128724 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:06.128737 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:06.128784 (+ 47us) server_negotiation.cc:407] Connection header received
0925 05:00:06.130433 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:06.130455 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:06.130897 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:06.131254 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:06.133079 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:06.134395 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:06.136236 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:06.137366 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:06.137513 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:06.142581 (+ 5068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:06.142589 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:06.142618 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:06.142659 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:06.145957 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:06.146831 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:06.146838 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:06.146849 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:06.147036 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:06.147530 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:06.147534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:06.147536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:06.148199 (+ 663us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:06.148547 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:06.149076 (+ 529us) server_negotiation.cc:299] Negotiation successful
0925 05:00:06.149461 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":422,"threads_started":1}
I20260925 05:00:06.152685 28089 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc627d7693e64c8cb58e5cdf140d6ecd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc627d7693e64c8cb58e5cdf140d6ecd" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33913 } }
I20260925 05:00:06.153213 28089 leader_election.cc:302] T 00000000000000000000000000000000 P bc627d7693e64c8cb58e5cdf140d6ecd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc627d7693e64c8cb58e5cdf140d6ecd; no voters:
I20260925 05:00:06.153908 28089 leader_election.cc:288] T 00000000000000000000000000000000 P bc627d7693e64c8cb58e5cdf140d6ecd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:00:06.154021 28094 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc627d7693e64c8cb58e5cdf140d6ecd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:00:06.155381 6262 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:00:06.155316 28094 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc627d7693e64c8cb58e5cdf140d6ecd [term 1 LEADER]: Becoming Leader. State: Replica: bc627d7693e64c8cb58e5cdf140d6ecd, State: Running, Role: LEADER
I20260925 05:00:06.155927 28094 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc627d7693e64c8cb58e5cdf140d6ecd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc627d7693e64c8cb58e5cdf140d6ecd" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33913 } }
I20260925 05:00:06.156276 28089 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc627d7693e64c8cb58e5cdf140d6ecd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:00:06.165365 28096 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc627d7693e64c8cb58e5cdf140d6ecd [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc627d7693e64c8cb58e5cdf140d6ecd. Latest consensus state: current_term: 1 leader_uuid: "bc627d7693e64c8cb58e5cdf140d6ecd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc627d7693e64c8cb58e5cdf140d6ecd" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33913 } } }
I20260925 05:00:06.165930 28096 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc627d7693e64c8cb58e5cdf140d6ecd [sys.catalog]: This master's current role is: LEADER
I20260925 05:00:06.165372 28095 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc627d7693e64c8cb58e5cdf140d6ecd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc627d7693e64c8cb58e5cdf140d6ecd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc627d7693e64c8cb58e5cdf140d6ecd" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 33913 } } }
I20260925 05:00:06.166323 28095 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc627d7693e64c8cb58e5cdf140d6ecd [sys.catalog]: This master's current role is: LEADER
I20260925 05:00:06.166785 28106 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:00:06.178992 28106 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:00:06.194427 28092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:06.161428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60111 (local address 127.6.29.190:33913)
0925 05:00:06.161654 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:06.161662 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:06.176998 (+ 15336us) server_negotiation.cc:407] Connection header received
0925 05:00:06.177223 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:06.177229 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:06.177322 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:06.177465 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:06.179660 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:06.180915 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:06.182701 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:06.183395 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:06.183552 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:06.186313 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:06.186318 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:06.186322 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:06.186352 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:06.191245 (+ 4893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:06.192264 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:06.192268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:06.192271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:06.192387 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:06.192873 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:06.192877 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:06.192879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:06.193190 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:06.193462 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:06.193860 (+ 398us) server_negotiation.cc:299] Negotiation successful
0925 05:00:06.194039 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 05:00:06.195732 28106 catalog_manager.cc:1347] Generated new cluster ID: 43db0dc6e77f435b887d55b74d5496da
I20260925 05:00:06.195880 28106 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:00:06.222216 28106 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:00:06.224134 28106 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:00:06.240360 28106 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc627d7693e64c8cb58e5cdf140d6ecd: Generated new TSK 0
I20260925 05:00:06.241228 28106 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:00:06.250350 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 28022
I20260925 05:00:06.251914 28084 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:00:06.346859 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 28022
W20260925 05:00:06.364439 28007 connection.cc:535] client connection to 127.6.29.190:33913 recv error: Network error: failed to read from TLS socket (remote: 127.6.29.190:33913): Connection reset by peer (error 104)
2026-09-25T05:00:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2043 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:00:06 dist-test-slave-xzjr krb5kdc[28121](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:00:06 dist-test-slave-xzjr krb5kdc[28121](info): set up 2 sockets
Sep 25 05:00:06 dist-test-slave-xzjr krb5kdc[28121](info): commencing operation
krb5kdc: starting...
W20260925 05:00:07.452623 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 05:00:07 dist-test-slave-xzjr krb5kdc[28121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:00:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:00:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.29.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
I20260925 05:00:07.724018 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:39479
--webserver_interface=127.6.29.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:32783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:39479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:00:08.057765 28134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:00:08.058084 28134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:00:08.058185 28134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:00:08.065425 28134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:00:08.065550 28134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:00:08.065598 28134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:00:08.065659 28134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:00:08.065697 28134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:00:08.075544 28134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:32783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:39479
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:39479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:00:08.076891 28134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:00:08.078187 28134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:00:08.090801 28142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:08.090728 28139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:08.091040 28140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:00:08.091621 28134 server_base.cc:1034] running on GCE node
I20260925 05:00:08.092548 28134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:00:08.094092 28134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:00:08.095321 28134 hybrid_clock.cc:648] HybridClock initialized: now 1790312408095252 us; error 58 us; skew 500 ppm
I20260925 05:00:08.095727 28134 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:00:08 dist-test-slave-xzjr krb5kdc[28121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312408, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:00:08.099284 28134 init.cc:377] Logged in from keytab as kudu/127.6.29.190@KRBTEST.COM (short username kudu)
I20260925 05:00:08.100996 28134 webserver.cc:460] Webserver started at http://127.6.29.190:34535/ using document root <none> and password file <none>
I20260925 05:00:08.101752 28134 fs_manager.cc:359] Metadata directory not provided
I20260925 05:00:08.101904 28134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:00:08.102238 28134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:00:08.105528 28134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "cabbaeb29b15421ba5f461459f34686f"
format_stamp: "Formatted at 2026-09-25 05:00:08 on dist-test-slave-xzjr"
server_key: "09cf603d67b3c2e46cca6588073d5dac"
server_key_iv: "9da43107afca18e3df8d0e19474f4b2f"
server_key_version: "encryptionkey@0"
I20260925 05:00:08.106345 28134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "cabbaeb29b15421ba5f461459f34686f"
format_stamp: "Formatted at 2026-09-25 05:00:08 on dist-test-slave-xzjr"
server_key: "09cf603d67b3c2e46cca6588073d5dac"
server_key_iv: "9da43107afca18e3df8d0e19474f4b2f"
server_key_version: "encryptionkey@0"
I20260925 05:00:08.111817 28134 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 05:00:08.115578 28149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:08.117115 28134 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 05:00:08.117347 28134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "cabbaeb29b15421ba5f461459f34686f"
format_stamp: "Formatted at 2026-09-25 05:00:08 on dist-test-slave-xzjr"
server_key: "09cf603d67b3c2e46cca6588073d5dac"
server_key_iv: "9da43107afca18e3df8d0e19474f4b2f"
server_key_version: "encryptionkey@0"
I20260925 05:00:08.117566 28134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:00:08.134826 28134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:00:08.139387 28134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:00:08.139684 28134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:00:08.158839 28134 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:39479
I20260925 05:00:08.158871 28200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:39479 every 8 connection(s)
I20260925 05:00:08.160450 28134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 05:00:08.164301 28201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:08.169719 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 28134
I20260925 05:00:08.169904 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 05:00:08.170390 6262 external_mini_cluster.cc:1428] Setting key 23e54a174d99e8ce46e04fa22d177786
I20260925 05:00:08.175527 28201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cabbaeb29b15421ba5f461459f34686f: Bootstrap starting.
I20260925 05:00:08.178743 28201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cabbaeb29b15421ba5f461459f34686f: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:08.179898 28201 log.cc:824] T 00000000000000000000000000000000 P cabbaeb29b15421ba5f461459f34686f: Log is configured to *not* fsync() on all Append() calls
I20260925 05:00:08.182749 28201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cabbaeb29b15421ba5f461459f34686f: No bootstrap required, opened a new log
I20260925 05:00:08.183969 28204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:08.172275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60983 (local address 127.6.29.190:39479)
0925 05:00:08.173162 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:08.173174 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:08.173231 (+ 57us) server_negotiation.cc:407] Connection header received
0925 05:00:08.174946 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:08.174970 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:08.175425 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:08.175855 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:08.176422 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:08.177736 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:08.179150 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:08.179950 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:08.180148 (+ 198us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:08.181501 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:08.181509 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:08.181523 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:00:08.182146 (+ 623us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:08.182227 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:00:08.182253 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:08.182425 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:08.182710 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 05:00:08.182906 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":426,"threads_started":1}
I20260925 05:00:08.187958 6262 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:00:08.190899 28201 raft_consensus.cc:348] T 00000000000000000000000000000000 P cabbaeb29b15421ba5f461459f34686f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cabbaeb29b15421ba5f461459f34686f" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39479 } }
I20260925 05:00:08.191367 28201 raft_consensus.cc:374] T 00000000000000000000000000000000 P cabbaeb29b15421ba5f461459f34686f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:00:08.191495 28201 raft_consensus.cc:729] T 00000000000000000000000000000000 P cabbaeb29b15421ba5f461459f34686f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cabbaeb29b15421ba5f461459f34686f, State: Initialized, Role: FOLLOWER
I20260925 05:00:08.192106 28201 consensus_queue.cc:260] T 00000000000000000000000000000000 P cabbaeb29b15421ba5f461459f34686f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cabbaeb29b15421ba5f461459f34686f" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39479 } }
I20260925 05:00:08.192369 28201 raft_consensus.cc:388] T 00000000000000000000000000000000 P cabbaeb29b15421ba5f461459f34686f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:00:08.192536 28201 raft_consensus.cc:482] T 00000000000000000000000000000000 P cabbaeb29b15421ba5f461459f34686f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:00:08.192734 28201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cabbaeb29b15421ba5f461459f34686f [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:08.195061 28201 raft_consensus.cc:504] T 00000000000000000000000000000000 P cabbaeb29b15421ba5f461459f34686f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cabbaeb29b15421ba5f461459f34686f" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39479 } }
I20260925 05:00:08.195695 28201 leader_election.cc:302] T 00000000000000000000000000000000 P cabbaeb29b15421ba5f461459f34686f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cabbaeb29b15421ba5f461459f34686f; no voters:
I20260925 05:00:08.196403 28201 leader_election.cc:288] T 00000000000000000000000000000000 P cabbaeb29b15421ba5f461459f34686f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:00:08.196527 28211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cabbaeb29b15421ba5f461459f34686f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:00:08.197206 28211 raft_consensus.cc:686] T 00000000000000000000000000000000 P cabbaeb29b15421ba5f461459f34686f [term 1 LEADER]: Becoming Leader. State: Replica: cabbaeb29b15421ba5f461459f34686f, State: Running, Role: LEADER
I20260925 05:00:08.197878 28211 consensus_queue.cc:237] T 00000000000000000000000000000000 P cabbaeb29b15421ba5f461459f34686f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cabbaeb29b15421ba5f461459f34686f" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39479 } }
I20260925 05:00:08.200589 28212 sys_catalog.cc:455] T 00000000000000000000000000000000 P cabbaeb29b15421ba5f461459f34686f [sys.catalog]: SysCatalogTable state changed. Reason: New leader cabbaeb29b15421ba5f461459f34686f. Latest consensus state: current_term: 1 leader_uuid: "cabbaeb29b15421ba5f461459f34686f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cabbaeb29b15421ba5f461459f34686f" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39479 } } }
I20260925 05:00:08.201035 28212 sys_catalog.cc:458] T 00000000000000000000000000000000 P cabbaeb29b15421ba5f461459f34686f [sys.catalog]: This master's current role is: LEADER
I20260925 05:00:08.201319 28213 sys_catalog.cc:455] T 00000000000000000000000000000000 P cabbaeb29b15421ba5f461459f34686f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cabbaeb29b15421ba5f461459f34686f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cabbaeb29b15421ba5f461459f34686f" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 39479 } } }
I20260925 05:00:08.201618 28213 sys_catalog.cc:458] T 00000000000000000000000000000000 P cabbaeb29b15421ba5f461459f34686f [sys.catalog]: This master's current role is: LEADER
I20260925 05:00:08.201947 28204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:08.192663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35405 (local address 127.6.29.190:39479)
0925 05:00:08.192916 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:08.192925 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:08.193523 (+ 598us) server_negotiation.cc:407] Connection header received
0925 05:00:08.193761 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:08.193766 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:08.193835 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:08.193954 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:08.194469 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:08.195231 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:08.197422 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:08.198459 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:08.198588 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:08.200476 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:08.200486 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:08.200493 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:00:08.200520 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:08.200576 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:00:08.200623 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:08.200751 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:08.201364 (+ 613us) server_negotiation.cc:299] Negotiation successful
0925 05:00:08.201428 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 05:00:08.202419 28201 sys_catalog.cc:564] T 00000000000000000000000000000000 P cabbaeb29b15421ba5f461459f34686f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:00:08.212777 28215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:00:08.219609 28215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:00:08.229553 28215 catalog_manager.cc:1347] Generated new cluster ID: a74d043eaa494c7781828ab418224fa4
I20260925 05:00:08.229682 28215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:00:08.252175 28215 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:00:08.253510 28215 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:00:08.259459 28215 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cabbaeb29b15421ba5f461459f34686f: Generated new TSK 0
I20260925 05:00:08.260448 28215 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:00:08.269881 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 28134
W20260925 05:00:08.271101 28196 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60983
W20260925 05:00:08.271962 6262 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 28134: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:00:08.272122 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 28134
W20260925 05:00:08.289865 28119 connection.cc:535] client connection to 127.6.29.190:39479 recv error: Network error: failed to read from TLS socket (remote: 127.6.29.190:39479): Connection reset by peer (error 104)
2026-09-25T05:00:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1925 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:00:08 dist-test-slave-xzjr krb5kdc[28232](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:00:08 dist-test-slave-xzjr krb5kdc[28232](info): set up 2 sockets
Sep 25 05:00:08 dist-test-slave-xzjr krb5kdc[28232](info): commencing operation
krb5kdc: starting...
W20260925 05:00:09.380683 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 05:00:09 dist-test-slave-xzjr krb5kdc[28232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312409, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:00:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:00:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.29.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
I20260925 05:00:09.647027 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:34835
--webserver_interface=127.6.29.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:32811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:34835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:00:09.997252 28245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:00:09.997565 28245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:00:09.997669 28245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:00:10.008641 28245 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:00:10.008849 28245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:00:10.008955 28245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:00:10.009063 28245 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:00:10.009147 28245 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:00:10.021198 28245 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:32811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:34835
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:34835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:00:10.022581 28245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:00:10.024009 28245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:00:10.036209 28250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:10.036248 28253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:10.036268 28251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:00:10.036903 28245 server_base.cc:1034] running on GCE node
I20260925 05:00:10.037737 28245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:00:10.039130 28245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:00:10.040364 28245 hybrid_clock.cc:648] HybridClock initialized: now 1790312410040346 us; error 45 us; skew 500 ppm
I20260925 05:00:10.040725 28245 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:00:10 dist-test-slave-xzjr krb5kdc[28232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312410, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:00:10.043934 28245 init.cc:377] Logged in from keytab as kudu/127.6.29.190@KRBTEST.COM (short username kudu)
I20260925 05:00:10.045486 28245 webserver.cc:460] Webserver started at http://127.6.29.190:41527/ using document root <none> and password file <none>
I20260925 05:00:10.046177 28245 fs_manager.cc:359] Metadata directory not provided
I20260925 05:00:10.046321 28245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:00:10.046636 28245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:00:10.049744 28245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "4beacad231f74df8aa082cf098058ed1"
format_stamp: "Formatted at 2026-09-25 05:00:10 on dist-test-slave-xzjr"
server_key: "68915603e473c23ba164de15d2030398"
server_key_iv: "381c517925901c9c2e849c788846161e"
server_key_version: "encryptionkey@0"
I20260925 05:00:10.050736 28245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "4beacad231f74df8aa082cf098058ed1"
format_stamp: "Formatted at 2026-09-25 05:00:10 on dist-test-slave-xzjr"
server_key: "68915603e473c23ba164de15d2030398"
server_key_iv: "381c517925901c9c2e849c788846161e"
server_key_version: "encryptionkey@0"
I20260925 05:00:10.056205 28245 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 05:00:10.059734 28260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:10.061142 28245 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 05:00:10.061336 28245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "4beacad231f74df8aa082cf098058ed1"
format_stamp: "Formatted at 2026-09-25 05:00:10 on dist-test-slave-xzjr"
server_key: "68915603e473c23ba164de15d2030398"
server_key_iv: "381c517925901c9c2e849c788846161e"
server_key_version: "encryptionkey@0"
I20260925 05:00:10.061542 28245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:00:10.067346 28245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:00:10.071856 28245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:00:10.072160 28245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:00:10.091024 28245 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:34835
I20260925 05:00:10.091066 28311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:34835 every 8 connection(s)
I20260925 05:00:10.092514 28245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 05:00:10.093111 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 28245
I20260925 05:00:10.093317 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 05:00:10.093919 6262 external_mini_cluster.cc:1428] Setting key 42bb7c29ce59e8118b4ef43ff82929b2
I20260925 05:00:10.096915 28312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:10.101745 28315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:10.095729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58095 (local address 127.6.29.190:34835)
0925 05:00:10.096576 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:10.096588 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:10.096642 (+ 54us) server_negotiation.cc:407] Connection header received
0925 05:00:10.098336 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:10.098359 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:10.098701 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:10.099023 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:10.100141 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:10.100148 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:10.100161 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:00:10.100178 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:10.100252 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:00:10.100272 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:10.100439 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:10.100562 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 05:00:10.100801 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":344,"threads_started":1}
I20260925 05:00:10.105792 6262 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:00:10.109347 28312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4beacad231f74df8aa082cf098058ed1: Bootstrap starting.
I20260925 05:00:10.113131 28315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:10.110439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60363 (local address 127.6.29.190:34835)
0925 05:00:10.110668 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:10.110675 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:10.111032 (+ 357us) server_negotiation.cc:407] Connection header received
0925 05:00:10.111230 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:10.111234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:10.111285 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:10.111376 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:10.112420 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:10.112424 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:10.112428 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:00:10.112444 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:10.112484 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:00:10.112499 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:10.112595 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:10.112752 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 05:00:10.112806 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 05:00:10.113540 28312 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4beacad231f74df8aa082cf098058ed1: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:10.114686 28312 log.cc:824] T 00000000000000000000000000000000 P 4beacad231f74df8aa082cf098058ed1: Log is configured to *not* fsync() on all Append() calls
I20260925 05:00:10.117612 28312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4beacad231f74df8aa082cf098058ed1: No bootstrap required, opened a new log
I20260925 05:00:10.125108 28312 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4beacad231f74df8aa082cf098058ed1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4beacad231f74df8aa082cf098058ed1" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 34835 } }
I20260925 05:00:10.125551 28312 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4beacad231f74df8aa082cf098058ed1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:00:10.125689 28312 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4beacad231f74df8aa082cf098058ed1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4beacad231f74df8aa082cf098058ed1, State: Initialized, Role: FOLLOWER
I20260925 05:00:10.126406 28312 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4beacad231f74df8aa082cf098058ed1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4beacad231f74df8aa082cf098058ed1" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 34835 } }
I20260925 05:00:10.126708 28312 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4beacad231f74df8aa082cf098058ed1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:00:10.126860 28312 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4beacad231f74df8aa082cf098058ed1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:00:10.127051 28312 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4beacad231f74df8aa082cf098058ed1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:10.129390 28312 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4beacad231f74df8aa082cf098058ed1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4beacad231f74df8aa082cf098058ed1" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 34835 } }
I20260925 05:00:10.129927 28312 leader_election.cc:302] T 00000000000000000000000000000000 P 4beacad231f74df8aa082cf098058ed1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4beacad231f74df8aa082cf098058ed1; no voters:
I20260925 05:00:10.130571 28312 leader_election.cc:288] T 00000000000000000000000000000000 P 4beacad231f74df8aa082cf098058ed1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:00:10.130743 28322 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4beacad231f74df8aa082cf098058ed1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:00:10.131450 28322 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4beacad231f74df8aa082cf098058ed1 [term 1 LEADER]: Becoming Leader. State: Replica: 4beacad231f74df8aa082cf098058ed1, State: Running, Role: LEADER
I20260925 05:00:10.132254 28322 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4beacad231f74df8aa082cf098058ed1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4beacad231f74df8aa082cf098058ed1" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 34835 } }
I20260925 05:00:10.133024 28312 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4beacad231f74df8aa082cf098058ed1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:00:10.135537 28324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4beacad231f74df8aa082cf098058ed1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4beacad231f74df8aa082cf098058ed1. Latest consensus state: current_term: 1 leader_uuid: "4beacad231f74df8aa082cf098058ed1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4beacad231f74df8aa082cf098058ed1" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 34835 } } }
I20260925 05:00:10.135993 28324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4beacad231f74df8aa082cf098058ed1 [sys.catalog]: This master's current role is: LEADER
I20260925 05:00:10.136178 28323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4beacad231f74df8aa082cf098058ed1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4beacad231f74df8aa082cf098058ed1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4beacad231f74df8aa082cf098058ed1" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 34835 } } }
I20260925 05:00:10.136497 28323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4beacad231f74df8aa082cf098058ed1 [sys.catalog]: This master's current role is: LEADER
I20260925 05:00:10.137686 28331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:00:10.143136 28331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:00:10.152015 28331 catalog_manager.cc:1347] Generated new cluster ID: 5af329e5e7e14d51b04180b869ea4e22
I20260925 05:00:10.152149 28331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:00:10.173272 28331 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:00:10.173750 28331 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:00:10.192278 28331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4beacad231f74df8aa082cf098058ed1: Generated new TSK 0
I20260925 05:00:10.193197 28331 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:00:10.199447 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 28245
W20260925 05:00:10.200810 28307 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58095
W20260925 05:00:10.201572 6262 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 28245: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:00:10.201725 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 28245
2026-09-25T05:00:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1926 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:00:10 dist-test-slave-xzjr krb5kdc[28343](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:00:10 dist-test-slave-xzjr krb5kdc[28343](info): set up 2 sockets
Sep 25 05:00:10 dist-test-slave-xzjr krb5kdc[28343](info): commencing operation
krb5kdc: starting...
W20260925 05:00:11.312634 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 05:00:11 dist-test-slave-xzjr krb5kdc[28343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:00:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:00:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.29.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
I20260925 05:00:11.579155 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:41583
--webserver_interface=127.6.29.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:41827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:41583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:00:11.915709 28356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:00:11.916033 28356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:00:11.916234 28356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:00:11.923599 28356 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:00:11.923725 28356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:00:11.923844 28356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:00:11.923957 28356 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:00:11.924041 28356 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:00:11.933924 28356 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:41827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:41583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:41583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:00:11.935299 28356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:00:11.936625 28356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:00:11.948652 28362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:11.948679 28361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:11.948738 28364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:00:11.949596 28356 server_base.cc:1034] running on GCE node
I20260925 05:00:11.950344 28356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:00:11.951689 28356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:00:11.952891 28356 hybrid_clock.cc:648] HybridClock initialized: now 1790312411952854 us; error 25 us; skew 500 ppm
I20260925 05:00:11.953261 28356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:00:11 dist-test-slave-xzjr krb5kdc[28343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312411, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:00:11.956499 28356 init.cc:377] Logged in from keytab as kudu/127.6.29.190@KRBTEST.COM (short username kudu)
I20260925 05:00:11.958024 28356 webserver.cc:460] Webserver started at http://127.6.29.190:34851/ using document root <none> and password file <none>
I20260925 05:00:11.958715 28356 fs_manager.cc:359] Metadata directory not provided
I20260925 05:00:11.958853 28356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:00:11.959167 28356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:00:11.962224 28356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "ef02c518e9e74d2a8a5f3103ae51b651"
format_stamp: "Formatted at 2026-09-25 05:00:11 on dist-test-slave-xzjr"
server_key: "e8918f403e597e0be7edca79aa58943d"
server_key_iv: "2e0a220dce5f01dc2bee1650a77c56dd"
server_key_version: "encryptionkey@0"
I20260925 05:00:11.963016 28356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "ef02c518e9e74d2a8a5f3103ae51b651"
format_stamp: "Formatted at 2026-09-25 05:00:11 on dist-test-slave-xzjr"
server_key: "e8918f403e597e0be7edca79aa58943d"
server_key_iv: "2e0a220dce5f01dc2bee1650a77c56dd"
server_key_version: "encryptionkey@0"
I20260925 05:00:11.968472 28356 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 05:00:11.972165 28371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:11.973613 28356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 05:00:11.973842 28356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "ef02c518e9e74d2a8a5f3103ae51b651"
format_stamp: "Formatted at 2026-09-25 05:00:11 on dist-test-slave-xzjr"
server_key: "e8918f403e597e0be7edca79aa58943d"
server_key_iv: "2e0a220dce5f01dc2bee1650a77c56dd"
server_key_version: "encryptionkey@0"
I20260925 05:00:11.974066 28356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:00:11.986106 28356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:00:11.990283 28356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:00:11.990578 28356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:00:12.009322 28356 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:41583
I20260925 05:00:12.009356 28422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:41583 every 8 connection(s)
I20260925 05:00:12.010772 28356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 05:00:12.014724 28423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:12.015323 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 28356
I20260925 05:00:12.015524 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 05:00:12.016032 6262 external_mini_cluster.cc:1428] Setting key c2bba56a14735421cdc7e0538072be17
I20260925 05:00:12.028991 28423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef02c518e9e74d2a8a5f3103ae51b651: Bootstrap starting.
Sep 25 05:00:12 dist-test-slave-xzjr krb5kdc[28343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 05:00:12.032723 28423 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef02c518e9e74d2a8a5f3103ae51b651: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:12.034004 28423 log.cc:824] T 00000000000000000000000000000000 P ef02c518e9e74d2a8a5f3103ae51b651: Log is configured to *not* fsync() on all Append() calls
I20260925 05:00:12.037086 28423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef02c518e9e74d2a8a5f3103ae51b651: No bootstrap required, opened a new log
I20260925 05:00:12.039834 28426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:12.017895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58879 (local address 127.6.29.190:41583)
0925 05:00:12.018772 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:12.018784 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:12.018829 (+ 45us) server_negotiation.cc:407] Connection header received
0925 05:00:12.020746 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:12.020771 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:12.021189 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:12.021524 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:12.023272 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:12.024739 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:12.026615 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:12.027290 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:12.027417 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:12.032128 (+ 4711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:12.032136 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:12.032152 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:12.032198 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:12.035481 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:12.036305 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:12.036313 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:12.036319 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:12.036487 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:12.036889 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:12.036894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:12.036898 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:12.037475 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:12.037792 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:12.038287 (+ 495us) server_negotiation.cc:299] Negotiation successful
0925 05:00:12.038733 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":388,"threads_started":1}
I20260925 05:00:12.043915 6262 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:00:12.045838 28423 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef02c518e9e74d2a8a5f3103ae51b651 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef02c518e9e74d2a8a5f3103ae51b651" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 41583 } }
I20260925 05:00:12.046205 28423 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef02c518e9e74d2a8a5f3103ae51b651 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:00:12.046331 28423 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef02c518e9e74d2a8a5f3103ae51b651 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef02c518e9e74d2a8a5f3103ae51b651, State: Initialized, Role: FOLLOWER
I20260925 05:00:12.047075 28423 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef02c518e9e74d2a8a5f3103ae51b651 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef02c518e9e74d2a8a5f3103ae51b651" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 41583 } }
I20260925 05:00:12.047330 28423 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef02c518e9e74d2a8a5f3103ae51b651 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:00:12.047469 28423 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef02c518e9e74d2a8a5f3103ae51b651 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:00:12.047652 28423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef02c518e9e74d2a8a5f3103ae51b651 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:12.051165 28423 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef02c518e9e74d2a8a5f3103ae51b651 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef02c518e9e74d2a8a5f3103ae51b651" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 41583 } }
I20260925 05:00:12.051817 28423 leader_election.cc:302] T 00000000000000000000000000000000 P ef02c518e9e74d2a8a5f3103ae51b651 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef02c518e9e74d2a8a5f3103ae51b651; no voters:
I20260925 05:00:12.052500 28423 leader_election.cc:288] T 00000000000000000000000000000000 P ef02c518e9e74d2a8a5f3103ae51b651 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:00:12.052565 28433 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef02c518e9e74d2a8a5f3103ae51b651 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:00:12.054097 28433 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef02c518e9e74d2a8a5f3103ae51b651 [term 1 LEADER]: Becoming Leader. State: Replica: ef02c518e9e74d2a8a5f3103ae51b651, State: Running, Role: LEADER
I20260925 05:00:12.054896 28423 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef02c518e9e74d2a8a5f3103ae51b651 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:00:12.054816 28433 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef02c518e9e74d2a8a5f3103ae51b651 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef02c518e9e74d2a8a5f3103ae51b651" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 41583 } }
I20260925 05:00:12.065003 28434 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef02c518e9e74d2a8a5f3103ae51b651 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef02c518e9e74d2a8a5f3103ae51b651" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef02c518e9e74d2a8a5f3103ae51b651" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 41583 } } }
I20260925 05:00:12.065461 28434 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef02c518e9e74d2a8a5f3103ae51b651 [sys.catalog]: This master's current role is: LEADER
I20260925 05:00:12.067557 28426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:12.048410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37087 (local address 127.6.29.190:41583)
0925 05:00:12.048655 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:12.048663 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:12.049031 (+ 368us) server_negotiation.cc:407] Connection header received
0925 05:00:12.049216 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:12.049220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:12.049283 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:12.049370 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:12.051245 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:12.052160 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:12.053967 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:12.054855 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:12.055022 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:12.058473 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:12.058482 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:12.058491 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:12.058547 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:12.063427 (+ 4880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:12.064316 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:12.064321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:12.064325 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:12.064475 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:12.065008 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:12.065017 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:12.065023 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:12.065625 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:12.065910 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:12.066686 (+ 776us) server_negotiation.cc:299] Negotiation successful
0925 05:00:12.067013 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 05:00:12.069545 28435 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef02c518e9e74d2a8a5f3103ae51b651 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef02c518e9e74d2a8a5f3103ae51b651. Latest consensus state: current_term: 1 leader_uuid: "ef02c518e9e74d2a8a5f3103ae51b651" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef02c518e9e74d2a8a5f3103ae51b651" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 41583 } } }
I20260925 05:00:12.069890 28435 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef02c518e9e74d2a8a5f3103ae51b651 [sys.catalog]: This master's current role is: LEADER
I20260925 05:00:12.070968 28444 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:00:12.075748 28444 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:00:12.084720 28444 catalog_manager.cc:1347] Generated new cluster ID: fcd09ca8a3e64a0cacce3a7592b3b8c8
I20260925 05:00:12.084857 28444 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:00:12.096679 28444 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:00:12.098207 28444 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:00:12.112443 28444 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef02c518e9e74d2a8a5f3103ae51b651: Generated new TSK 0
I20260925 05:00:12.113411 28444 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:00:12.126433 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 28356
I20260925 05:00:12.127864 28418 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:00:12.225625 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 28356
2026-09-25T05:00:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2026 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:00:12 dist-test-slave-xzjr krb5kdc[28456](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:00:12 dist-test-slave-xzjr krb5kdc[28456](info): set up 2 sockets
Sep 25 05:00:12 dist-test-slave-xzjr krb5kdc[28456](info): commencing operation
krb5kdc: starting...
W20260925 05:00:13.332657 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 05:00:13 dist-test-slave-xzjr krb5kdc[28456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:00:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:00:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.29.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
I20260925 05:00:13.595039 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:38897
--webserver_interface=127.6.29.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:33331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:38897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:00:13.930069 28469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:00:13.930438 28469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:00:13.930620 28469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:00:13.937914 28469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:00:13.938090 28469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:00:13.938225 28469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:00:13.938335 28469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:00:13.938426 28469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:00:13.948454 28469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:33331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:38897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:38897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:00:13.949852 28469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:00:13.951256 28469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:00:13.963544 28474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:13.963584 28475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:13.963791 28477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:00:13.964555 28469 server_base.cc:1034] running on GCE node
I20260925 05:00:13.965408 28469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:00:13.966862 28469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:00:13.968070 28469 hybrid_clock.cc:648] HybridClock initialized: now 1790312413968018 us; error 40 us; skew 500 ppm
I20260925 05:00:13.968454 28469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:00:13 dist-test-slave-xzjr krb5kdc[28456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312413, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:00:13.971930 28469 init.cc:377] Logged in from keytab as kudu/127.6.29.190@KRBTEST.COM (short username kudu)
I20260925 05:00:13.973457 28469 webserver.cc:460] Webserver started at http://127.6.29.190:44095/ using document root <none> and password file <none>
I20260925 05:00:13.974192 28469 fs_manager.cc:359] Metadata directory not provided
I20260925 05:00:13.974325 28469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:00:13.974645 28469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:00:13.977849 28469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "aadb379d780a432fab890c0769b8062c"
format_stamp: "Formatted at 2026-09-25 05:00:13 on dist-test-slave-xzjr"
server_key: "a4debb70aba93a4df1bb40448ac5ce3e"
server_key_iv: "8a92ead232a795b37d8e3c62f75a523b"
server_key_version: "encryptionkey@0"
I20260925 05:00:13.978735 28469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "aadb379d780a432fab890c0769b8062c"
format_stamp: "Formatted at 2026-09-25 05:00:13 on dist-test-slave-xzjr"
server_key: "a4debb70aba93a4df1bb40448ac5ce3e"
server_key_iv: "8a92ead232a795b37d8e3c62f75a523b"
server_key_version: "encryptionkey@0"
I20260925 05:00:13.984346 28469 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 05:00:13.988224 28484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:13.989825 28469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 05:00:13.990021 28469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "aadb379d780a432fab890c0769b8062c"
format_stamp: "Formatted at 2026-09-25 05:00:13 on dist-test-slave-xzjr"
server_key: "a4debb70aba93a4df1bb40448ac5ce3e"
server_key_iv: "8a92ead232a795b37d8e3c62f75a523b"
server_key_version: "encryptionkey@0"
I20260925 05:00:13.990243 28469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:00:14.005679 28469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:00:14.010288 28469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:00:14.010582 28469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:00:14.029639 28469 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:38897
I20260925 05:00:14.029675 28535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:38897 every 8 connection(s)
I20260925 05:00:14.031042 28469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 05:00:14.034895 28536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:14.040719 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 28469
I20260925 05:00:14.040908 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 05:00:14.041477 6262 external_mini_cluster.cc:1428] Setting key 8ef4915a81831067db916a6ea0efe414
I20260925 05:00:14.046401 28536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aadb379d780a432fab890c0769b8062c: Bootstrap starting.
I20260925 05:00:14.050168 28536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aadb379d780a432fab890c0769b8062c: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:14.051330 28536 log.cc:824] T 00000000000000000000000000000000 P aadb379d780a432fab890c0769b8062c: Log is configured to *not* fsync() on all Append() calls
I20260925 05:00:14.054657 28536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aadb379d780a432fab890c0769b8062c: No bootstrap required, opened a new log
I20260925 05:00:14.055454 28539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:14.043350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57749 (local address 127.6.29.190:38897)
0925 05:00:14.044290 (+ 940us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:14.044302 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:14.044348 (+ 46us) server_negotiation.cc:407] Connection header received
0925 05:00:14.046024 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:14.046051 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:14.046461 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:14.046845 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:14.047460 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:14.048891 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:14.050761 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:14.051557 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:14.051686 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:14.053074 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:14.053082 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:14.053096 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:00:14.053640 (+ 544us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:14.053712 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:00:14.053737 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:14.053825 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:14.054069 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 05:00:14.054345 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":476,"threads_started":1}
I20260925 05:00:14.059268 6262 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:00:14.063036 28536 raft_consensus.cc:348] T 00000000000000000000000000000000 P aadb379d780a432fab890c0769b8062c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aadb379d780a432fab890c0769b8062c" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 38897 } }
I20260925 05:00:14.063580 28536 raft_consensus.cc:374] T 00000000000000000000000000000000 P aadb379d780a432fab890c0769b8062c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:00:14.063709 28536 raft_consensus.cc:729] T 00000000000000000000000000000000 P aadb379d780a432fab890c0769b8062c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aadb379d780a432fab890c0769b8062c, State: Initialized, Role: FOLLOWER
I20260925 05:00:14.064610 28536 consensus_queue.cc:260] T 00000000000000000000000000000000 P aadb379d780a432fab890c0769b8062c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aadb379d780a432fab890c0769b8062c" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 38897 } }
I20260925 05:00:14.064914 28536 raft_consensus.cc:388] T 00000000000000000000000000000000 P aadb379d780a432fab890c0769b8062c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:00:14.065084 28536 raft_consensus.cc:482] T 00000000000000000000000000000000 P aadb379d780a432fab890c0769b8062c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:00:14.065284 28536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aadb379d780a432fab890c0769b8062c [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:14.068085 28536 raft_consensus.cc:504] T 00000000000000000000000000000000 P aadb379d780a432fab890c0769b8062c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aadb379d780a432fab890c0769b8062c" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 38897 } }
I20260925 05:00:14.068697 28536 leader_election.cc:302] T 00000000000000000000000000000000 P aadb379d780a432fab890c0769b8062c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aadb379d780a432fab890c0769b8062c; no voters:
I20260925 05:00:14.069581 28536 leader_election.cc:288] T 00000000000000000000000000000000 P aadb379d780a432fab890c0769b8062c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:00:14.069741 28546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aadb379d780a432fab890c0769b8062c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:00:14.070622 28539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:14.063647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37037 (local address 127.6.29.190:38897)
0925 05:00:14.063876 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:14.063884 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:14.064344 (+ 460us) server_negotiation.cc:407] Connection header received
0925 05:00:14.064534 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:14.064539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:14.064629 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:14.064739 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:14.065284 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:14.066420 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:14.067721 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:14.068416 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:14.068559 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:14.069724 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:14.069730 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:14.069735 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:00:14.069758 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:14.069803 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:00:14.069826 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:14.069929 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:14.070177 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 05:00:14.070285 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":86}
I20260925 05:00:14.071434 28546 raft_consensus.cc:686] T 00000000000000000000000000000000 P aadb379d780a432fab890c0769b8062c [term 1 LEADER]: Becoming Leader. State: Replica: aadb379d780a432fab890c0769b8062c, State: Running, Role: LEADER
I20260925 05:00:14.072047 28536 sys_catalog.cc:564] T 00000000000000000000000000000000 P aadb379d780a432fab890c0769b8062c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:00:14.072080 28546 consensus_queue.cc:237] T 00000000000000000000000000000000 P aadb379d780a432fab890c0769b8062c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aadb379d780a432fab890c0769b8062c" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 38897 } }
I20260925 05:00:14.074973 28547 sys_catalog.cc:455] T 00000000000000000000000000000000 P aadb379d780a432fab890c0769b8062c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aadb379d780a432fab890c0769b8062c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aadb379d780a432fab890c0769b8062c" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 38897 } } }
I20260925 05:00:14.075682 28547 sys_catalog.cc:458] T 00000000000000000000000000000000 P aadb379d780a432fab890c0769b8062c [sys.catalog]: This master's current role is: LEADER
I20260925 05:00:14.075431 28548 sys_catalog.cc:455] T 00000000000000000000000000000000 P aadb379d780a432fab890c0769b8062c [sys.catalog]: SysCatalogTable state changed. Reason: New leader aadb379d780a432fab890c0769b8062c. Latest consensus state: current_term: 1 leader_uuid: "aadb379d780a432fab890c0769b8062c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aadb379d780a432fab890c0769b8062c" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 38897 } } }
I20260925 05:00:14.075855 28548 sys_catalog.cc:458] T 00000000000000000000000000000000 P aadb379d780a432fab890c0769b8062c [sys.catalog]: This master's current role is: LEADER
I20260925 05:00:14.076584 28555 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:00:14.083288 28555 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:00:14.092432 28555 catalog_manager.cc:1347] Generated new cluster ID: c23fcfde423148739716348db1baf3c6
I20260925 05:00:14.092547 28555 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:00:14.108003 28555 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:00:14.109380 28555 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:00:14.116619 28555 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aadb379d780a432fab890c0769b8062c: Generated new TSK 0
I20260925 05:00:14.117486 28555 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:00:14.128327 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 28469
W20260925 05:00:14.129570 28531 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57749
W20260925 05:00:14.130375 6262 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 28469: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:00:14.130548 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 28469
2026-09-25T05:00:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1904 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:00:14 dist-test-slave-xzjr krb5kdc[28567](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:00:14 dist-test-slave-xzjr krb5kdc[28567](info): set up 2 sockets
Sep 25 05:00:14 dist-test-slave-xzjr krb5kdc[28567](info): commencing operation
krb5kdc: starting...
W20260925 05:00:15.268852 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 05:00:15 dist-test-slave-xzjr krb5kdc[28567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:00:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:00:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.29.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
I20260925 05:00:15.563241 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:43809
--webserver_interface=127.6.29.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:39515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:43809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:00:15.899430 28580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:00:15.899827 28580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:00:15.900002 28580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:00:15.907385 28580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:00:15.907553 28580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:00:15.907653 28580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:00:15.907792 28580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:00:15.907864 28580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:00:15.918169 28580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:39515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:43809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:43809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:00:15.919540 28580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:00:15.920939 28580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:00:15.933245 28585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:15.933300 28588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:15.933277 28586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:00:15.934608 28580 server_base.cc:1034] running on GCE node
I20260925 05:00:15.935478 28580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:00:15.937054 28580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:00:15.938303 28580 hybrid_clock.cc:648] HybridClock initialized: now 1790312415938237 us; error 62 us; skew 500 ppm
I20260925 05:00:15.938699 28580 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:00:15 dist-test-slave-xzjr krb5kdc[28567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312415, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:00:15.941871 28580 init.cc:377] Logged in from keytab as kudu/127.6.29.190@KRBTEST.COM (short username kudu)
I20260925 05:00:15.943274 28580 webserver.cc:460] Webserver started at http://127.6.29.190:43257/ using document root <none> and password file <none>
I20260925 05:00:15.944032 28580 fs_manager.cc:359] Metadata directory not provided
I20260925 05:00:15.944139 28580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:00:15.944387 28580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:00:15.947434 28580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "b130e4a1466d45e19ff3ad82b20ceb7f"
format_stamp: "Formatted at 2026-09-25 05:00:15 on dist-test-slave-xzjr"
server_key: "b1abde2b215df37aa01ba293dd858e73"
server_key_iv: "0b082ae3dff7faa3d5329f94034d805a"
server_key_version: "encryptionkey@0"
I20260925 05:00:15.948240 28580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "b130e4a1466d45e19ff3ad82b20ceb7f"
format_stamp: "Formatted at 2026-09-25 05:00:15 on dist-test-slave-xzjr"
server_key: "b1abde2b215df37aa01ba293dd858e73"
server_key_iv: "0b082ae3dff7faa3d5329f94034d805a"
server_key_version: "encryptionkey@0"
I20260925 05:00:15.953924 28580 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 05:00:15.957384 28595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:15.958793 28580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 05:00:15.958971 28580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "b130e4a1466d45e19ff3ad82b20ceb7f"
format_stamp: "Formatted at 2026-09-25 05:00:15 on dist-test-slave-xzjr"
server_key: "b1abde2b215df37aa01ba293dd858e73"
server_key_iv: "0b082ae3dff7faa3d5329f94034d805a"
server_key_version: "encryptionkey@0"
I20260925 05:00:15.959208 28580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:00:15.965044 28580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:00:15.969538 28580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:00:15.969839 28580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:00:15.991083 28580 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:43809
I20260925 05:00:15.991128 28646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:43809 every 8 connection(s)
I20260925 05:00:15.992761 28580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 05:00:15.997267 28647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:15.999640 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 28580
I20260925 05:00:15.999861 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 05:00:16.000327 6262 external_mini_cluster.cc:1428] Setting key 9b81f4010b77d9508a3188b9f7afa459
I20260925 05:00:16.009195 28650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:16.002535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37477 (local address 127.6.29.190:43809)
0925 05:00:16.003504 (+ 969us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:16.003517 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:16.003565 (+ 48us) server_negotiation.cc:407] Connection header received
0925 05:00:16.005294 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:16.005321 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:16.005742 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:16.006118 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:16.007438 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:16.007454 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:16.007468 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:00:16.007500 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:16.007571 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:00:16.007592 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:16.007861 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:16.007988 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 05:00:16.008240 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":444,"threads_started":1}
I20260925 05:00:16.012846 28647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b130e4a1466d45e19ff3ad82b20ceb7f: Bootstrap starting.
I20260925 05:00:16.013108 6262 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:00:16.016207 28647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b130e4a1466d45e19ff3ad82b20ceb7f: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:16.017428 28647 log.cc:824] T 00000000000000000000000000000000 P b130e4a1466d45e19ff3ad82b20ceb7f: Log is configured to *not* fsync() on all Append() calls
I20260925 05:00:16.020633 28650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:16.017743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58295 (local address 127.6.29.190:43809)
0925 05:00:16.017950 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:16.017959 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:16.018469 (+ 510us) server_negotiation.cc:407] Connection header received
0925 05:00:16.018681 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:16.018686 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:16.018754 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:16.018876 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:16.019938 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:16.019944 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:16.019949 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 05:00:16.019972 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:16.020023 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 05:00:16.020039 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:16.020177 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:16.020277 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 05:00:16.020348 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 05:00:16.021522 28647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b130e4a1466d45e19ff3ad82b20ceb7f: No bootstrap required, opened a new log
I20260925 05:00:16.029958 28647 raft_consensus.cc:348] T 00000000000000000000000000000000 P b130e4a1466d45e19ff3ad82b20ceb7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b130e4a1466d45e19ff3ad82b20ceb7f" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43809 } }
I20260925 05:00:16.030402 28647 raft_consensus.cc:374] T 00000000000000000000000000000000 P b130e4a1466d45e19ff3ad82b20ceb7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 05:00:16.030525 28647 raft_consensus.cc:729] T 00000000000000000000000000000000 P b130e4a1466d45e19ff3ad82b20ceb7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b130e4a1466d45e19ff3ad82b20ceb7f, State: Initialized, Role: FOLLOWER
I20260925 05:00:16.031186 28647 consensus_queue.cc:260] T 00000000000000000000000000000000 P b130e4a1466d45e19ff3ad82b20ceb7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b130e4a1466d45e19ff3ad82b20ceb7f" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43809 } }
I20260925 05:00:16.031441 28647 raft_consensus.cc:388] T 00000000000000000000000000000000 P b130e4a1466d45e19ff3ad82b20ceb7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 05:00:16.031599 28647 raft_consensus.cc:482] T 00000000000000000000000000000000 P b130e4a1466d45e19ff3ad82b20ceb7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 05:00:16.031821 28647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b130e4a1466d45e19ff3ad82b20ceb7f [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:16.034456 28647 raft_consensus.cc:504] T 00000000000000000000000000000000 P b130e4a1466d45e19ff3ad82b20ceb7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b130e4a1466d45e19ff3ad82b20ceb7f" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43809 } }
I20260925 05:00:16.035027 28647 leader_election.cc:302] T 00000000000000000000000000000000 P b130e4a1466d45e19ff3ad82b20ceb7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b130e4a1466d45e19ff3ad82b20ceb7f; no voters:
I20260925 05:00:16.035693 28647 leader_election.cc:288] T 00000000000000000000000000000000 P b130e4a1466d45e19ff3ad82b20ceb7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 05:00:16.035908 28657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b130e4a1466d45e19ff3ad82b20ceb7f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:00:16.036603 28657 raft_consensus.cc:686] T 00000000000000000000000000000000 P b130e4a1466d45e19ff3ad82b20ceb7f [term 1 LEADER]: Becoming Leader. State: Replica: b130e4a1466d45e19ff3ad82b20ceb7f, State: Running, Role: LEADER
I20260925 05:00:16.037178 28657 consensus_queue.cc:237] T 00000000000000000000000000000000 P b130e4a1466d45e19ff3ad82b20ceb7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b130e4a1466d45e19ff3ad82b20ceb7f" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43809 } }
I20260925 05:00:16.038017 28647 sys_catalog.cc:564] T 00000000000000000000000000000000 P b130e4a1466d45e19ff3ad82b20ceb7f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:00:16.040237 28659 sys_catalog.cc:455] T 00000000000000000000000000000000 P b130e4a1466d45e19ff3ad82b20ceb7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b130e4a1466d45e19ff3ad82b20ceb7f. Latest consensus state: current_term: 1 leader_uuid: "b130e4a1466d45e19ff3ad82b20ceb7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b130e4a1466d45e19ff3ad82b20ceb7f" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43809 } } }
I20260925 05:00:16.040719 28659 sys_catalog.cc:458] T 00000000000000000000000000000000 P b130e4a1466d45e19ff3ad82b20ceb7f [sys.catalog]: This master's current role is: LEADER
I20260925 05:00:16.041900 28658 sys_catalog.cc:455] T 00000000000000000000000000000000 P b130e4a1466d45e19ff3ad82b20ceb7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b130e4a1466d45e19ff3ad82b20ceb7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b130e4a1466d45e19ff3ad82b20ceb7f" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43809 } } }
I20260925 05:00:16.042281 28658 sys_catalog.cc:458] T 00000000000000000000000000000000 P b130e4a1466d45e19ff3ad82b20ceb7f [sys.catalog]: This master's current role is: LEADER
I20260925 05:00:16.043146 28665 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:00:16.048882 28665 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:00:16.058137 28665 catalog_manager.cc:1347] Generated new cluster ID: 1b50b1059d6f4efe9f5fdbc6a577ddbb
I20260925 05:00:16.058272 28665 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:00:16.072705 28665 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:00:16.073282 28665 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:00:16.085565 28665 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b130e4a1466d45e19ff3ad82b20ceb7f: Generated new TSK 0
I20260925 05:00:16.086549 28665 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 05:00:16.094481 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 28580
W20260925 05:00:16.095708 28642 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37477
W20260925 05:00:16.096596 6262 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 28580: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 05:00:16.096779 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 28580
2026-09-25T05:00:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1966 ms)
[----------] 18 tests from AuthTokenIssuingTest (35458 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:00:16 dist-test-slave-xzjr krb5kdc[28678](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:00:16 dist-test-slave-xzjr krb5kdc[28678](info): set up 2 sockets
Sep 25 05:00:16 dist-test-slave-xzjr krb5kdc[28678](info): commencing operation
krb5kdc: starting...
W20260925 05:00:17.212958 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 05:00:17 dist-test-slave-xzjr krb5kdc[28678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:00:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:00:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.29.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
I20260925 05:00:17.493319 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:40523
--webserver_interface=127.6.29.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:36155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:40523,127.6.29.189:36751,127.6.29.188:42981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:00:17.826548 28691 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 05:00:17.826916 28691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:00:17.827055 28691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:00:17.827195 28691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:00:17.834462 28691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:00:17.834563 28691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:00:17.834652 28691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:00:17.834731 28691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:00:17.834842 28691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:00:17.844856 28691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:36155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:40523,127.6.29.189:36751,127.6.29.188:42981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:40523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:00:17.846220 28691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:00:17.847544 28691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:00:17.859719 28699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:17.859699 28697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:17.860249 28696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:00:17.860524 28691 server_base.cc:1034] running on GCE node
I20260925 05:00:17.861297 28691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:00:17.862682 28691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:00:17.863894 28691 hybrid_clock.cc:648] HybridClock initialized: now 1790312417863826 us; error 45 us; skew 500 ppm
I20260925 05:00:17.864280 28691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:00:17 dist-test-slave-xzjr krb5kdc[28678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312417, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:00:17.867401 28691 init.cc:377] Logged in from keytab as kudu/127.6.29.190@KRBTEST.COM (short username kudu)
I20260925 05:00:17.868949 28691 webserver.cc:460] Webserver started at http://127.6.29.190:40223/ using document root <none> and password file <none>
I20260925 05:00:17.869704 28691 fs_manager.cc:359] Metadata directory not provided
I20260925 05:00:17.869843 28691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:00:17.870177 28691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:00:17.873270 28691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "3efa36f01dce409a897f14c46709858b"
format_stamp: "Formatted at 2026-09-25 05:00:17 on dist-test-slave-xzjr"
server_key: "62c41352f40b88fbbb45095787d33781"
server_key_iv: "c34691278bc59815c826cd636ca7377e"
server_key_version: "encryptionkey@0"
I20260925 05:00:17.874056 28691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "3efa36f01dce409a897f14c46709858b"
format_stamp: "Formatted at 2026-09-25 05:00:17 on dist-test-slave-xzjr"
server_key: "62c41352f40b88fbbb45095787d33781"
server_key_iv: "c34691278bc59815c826cd636ca7377e"
server_key_version: "encryptionkey@0"
I20260925 05:00:17.879254 28691 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 05:00:17.882848 28706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:17.884438 28691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:00:17.884631 28691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "3efa36f01dce409a897f14c46709858b"
format_stamp: "Formatted at 2026-09-25 05:00:17 on dist-test-slave-xzjr"
server_key: "62c41352f40b88fbbb45095787d33781"
server_key_iv: "c34691278bc59815c826cd636ca7377e"
server_key_version: "encryptionkey@0"
I20260925 05:00:17.884877 28691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:00:17.918440 28691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:00:17.922870 28691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:00:17.923166 28691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:00:17.942616 28691 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:40523
I20260925 05:00:17.942651 28757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:40523 every 8 connection(s)
I20260925 05:00:17.944103 28691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 05:00:17.947996 28758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:17.949223 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 28691
I20260925 05:00:17.949448 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 05:00:17.950284 6262 external_mini_cluster.cc:1428] Setting key 48ee3978de21a2d1916f237dadf91dab
I20260925 05:00:17.957855 28758 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40523 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:00:17 dist-test-slave-xzjr krb5kdc[28678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312417, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.29.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.189@KRBTEST.COM" created.
I20260925 05:00:17.981958 28762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:17.961879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:40523 (local address 127.0.0.1:34398)
0925 05:00:17.963109 (+ 1230us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:17.963143 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:17.964942 (+ 1799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:17.965771 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:17.965779 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:17.966477 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:00:17.966803 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:17.966821 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:17.968559 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:17.968566 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:17.969620 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:17.969632 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:17.970539 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:17.970545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:17.970715 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:17.971423 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:00:17.971445 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:00:17.973950 (+ 2505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:00:17.977320 (+ 3370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:17.977332 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:17.977344 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:17.977800 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:17.978276 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:17.978280 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:17.978282 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:17.978441 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:17.979334 (+ 893us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:00:17.979343 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:00:17.979582 (+ 239us) client_negotiation.cc:769] Sending connection context
0925 05:00:17.979905 (+ 323us) client_negotiation.cc:240] Negotiation successful
0925 05:00:17.980537 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1024,"mutex_wait_us":197,"thread_start_us":806,"threads_started":1}
I20260925 05:00:17.981981 28761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:17.961878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34398 (local address 127.6.29.190:40523)
0925 05:00:17.962810 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:17.962826 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:17.963264 (+ 438us) server_negotiation.cc:407] Connection header received
0925 05:00:17.965309 (+ 2045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:17.965315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:17.965539 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:17.965755 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:17.967013 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:17.968411 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:17.969771 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:17.970307 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:17.970503 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:17.974171 (+ 3668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:17.974179 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:17.974191 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:17.974217 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:17.977190 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:17.977946 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:17.977953 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:17.977956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:17.978165 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:17.978570 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:17.978577 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:17.978585 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:17.979185 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:17.979408 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:17.980017 (+ 609us) server_negotiation.cc:299] Negotiation successful
0925 05:00:17.980427 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":670,"thread_start_us":467,"threads_started":1}
I20260925 05:00:17.985180 28758 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.29.189" port: 36751 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:00:17.986395 28762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:17.986000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:36751 (local address 127.0.0.1:40742)
0925 05:00:17.986173 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:17.986277 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.189:36751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260925 05:00:17.986691 28709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.29.189:36751: connect: Connection refused (error 111)
I20260925 05:00:17.988417 28762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:17.988020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:36751 (local address 127.0.0.1:40758)
0925 05:00:17.988218 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:17.988286 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.189:36751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260925 05:00:17.988719 28758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.29.189:36751: Network error: Client connection negotiation failed: client connection to 127.6.29.189:36751: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.189.keytab.
Entry for principal kudu/127.6.29.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.189.keytab.
I20260925 05:00:18.011373 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.189:36751
--webserver_interface=127.6.29.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.189.keytab
--principal=kudu/127.6.29.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:36155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:40523,127.6.29.189:36751,127.6.29.188:42981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:00:18.038066 28758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.29.189" port: 36751 } attempt: 1
I20260925 05:00:18.039258 28762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:18.038816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:36751 (local address 127.0.0.1:40772)
0925 05:00:18.039013 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:18.039122 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.189:36751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260925 05:00:18.040357 28762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:18.039963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:36751 (local address 127.0.0.1:40788)
0925 05:00:18.040108 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:18.040189 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.189:36751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 05:00:18.040596 28758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.29.189:36751: Network error: Client connection negotiation failed: client connection to 127.6.29.189:36751: connect: Connection refused (error 111)
I20260925 05:00:18.108878 28758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.29.189" port: 36751 } attempt: 2
I20260925 05:00:18.110067 28762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:18.109593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:36751 (local address 127.0.0.1:40792)
0925 05:00:18.109825 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:18.109925 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.189:36751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260925 05:00:18.111119 28762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:18.110763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:36751 (local address 127.0.0.1:40796)
0925 05:00:18.110913 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:18.110987 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.189:36751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 05:00:18.111353 28758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.29.189:36751: Network error: Client connection negotiation failed: client connection to 127.6.29.189:36751: connect: Connection refused (error 111)
I20260925 05:00:18.202632 28758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.29.189" port: 36751 } attempt: 3
I20260925 05:00:18.203876 28762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:18.203360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:36751 (local address 127.0.0.1:40804)
0925 05:00:18.203579 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:18.203680 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.189:36751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260925 05:00:18.204941 28762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:18.204581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:36751 (local address 127.0.0.1:40816)
0925 05:00:18.204740 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:18.204806 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.189:36751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 05:00:18.205145 28758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.29.189:36751: Network error: Client connection negotiation failed: client connection to 127.6.29.189:36751: connect: Connection refused (error 111)
W20260925 05:00:18.339955 28765 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 05:00:18.340313 28765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:00:18.340451 28765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:00:18.340540 28765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:00:18.347667 28765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:00:18.347756 28765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:00:18.347886 28765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:00:18.347995 28765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:00:18.348081 28765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:00:18.348403 28758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.29.189" port: 36751 } attempt: 4
I20260925 05:00:18.349602 28762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:18.349040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:36751 (local address 127.0.0.1:40828)
0925 05:00:18.349370 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:18.349472 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.189:36751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
I20260925 05:00:18.350783 28762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:18.350395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:36751 (local address 127.0.0.1:40842)
0925 05:00:18.350562 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:18.350657 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.189:36751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260925 05:00:18.351017 28758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.29.189:36751: Network error: Client connection negotiation failed: client connection to 127.6.29.189:36751: connect: Connection refused (error 111)
I20260925 05:00:18.358487 28765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:36155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:40523,127.6.29.189:36751,127.6.29.188:42981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.189.keytab
--principal=kudu/127.6.29.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.189:36751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.29.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:00:18.359922 28765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:00:18.361272 28765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:00:18.373437 28770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:18.373499 28771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:18.373523 28773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:00:18.374738 28765 server_base.cc:1034] running on GCE node
I20260925 05:00:18.375597 28765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:00:18.377079 28765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:00:18.378432 28765 hybrid_clock.cc:648] HybridClock initialized: now 1790312418378340 us; error 56 us; skew 500 ppm
I20260925 05:00:18.378859 28765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:00:18 dist-test-slave-xzjr krb5kdc[28678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312418, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:00:18.382575 28765 init.cc:377] Logged in from keytab as kudu/127.6.29.189@KRBTEST.COM (short username kudu)
I20260925 05:00:18.384268 28765 webserver.cc:460] Webserver started at http://127.6.29.189:35589/ using document root <none> and password file <none>
I20260925 05:00:18.384977 28765 fs_manager.cc:359] Metadata directory not provided
I20260925 05:00:18.385061 28765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:00:18.385319 28765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:00:18.388262 28765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-1/data/instance:
uuid: "11b833178e1e4ee583a3e21b0d502304"
format_stamp: "Formatted at 2026-09-25 05:00:18 on dist-test-slave-xzjr"
server_key: "bbfee18631ad4707ecc2c42747bfd1e6"
server_key_iv: "3571726b5a539e57c6d3e2232e00789c"
server_key_version: "encryptionkey@0"
I20260925 05:00:18.388968 28765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-1/wal/instance:
uuid: "11b833178e1e4ee583a3e21b0d502304"
format_stamp: "Formatted at 2026-09-25 05:00:18 on dist-test-slave-xzjr"
server_key: "bbfee18631ad4707ecc2c42747bfd1e6"
server_key_iv: "3571726b5a539e57c6d3e2232e00789c"
server_key_version: "encryptionkey@0"
I20260925 05:00:18.393913 28765 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260925 05:00:18.397329 28780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:18.398691 28765 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 05:00:18.398876 28765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-1/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-1/wal
uuid: "11b833178e1e4ee583a3e21b0d502304"
format_stamp: "Formatted at 2026-09-25 05:00:18 on dist-test-slave-xzjr"
server_key: "bbfee18631ad4707ecc2c42747bfd1e6"
server_key_iv: "3571726b5a539e57c6d3e2232e00789c"
server_key_version: "encryptionkey@0"
I20260925 05:00:18.399091 28765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:00:18.420588 28765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:00:18.424901 28765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:00:18.425189 28765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:00:18.444046 28765 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.189:36751
I20260925 05:00:18.444058 28831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.189:36751 every 8 connection(s)
I20260925 05:00:18.445567 28765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-1/data/info.pb
I20260925 05:00:18.449479 28832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:18.455924 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 28765
I20260925 05:00:18.456166 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-1/wal/instance
I20260925 05:00:18.456591 6262 external_mini_cluster.cc:1428] Setting key 91d4cbac1b876d2dc6e8ee0d6d95fbcc
I20260925 05:00:18.456172 28832 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40523 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:00:18 dist-test-slave-xzjr krb5kdc[28678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312418, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.189@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 05:00:18.478251 28761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:18.459061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34410 (local address 127.6.29.190:40523)
0925 05:00:18.459260 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:18.459264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:18.460647 (+ 1383us) server_negotiation.cc:407] Connection header received
0925 05:00:18.462618 (+ 1971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:18.462624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:18.462718 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:18.462834 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:18.464505 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:18.465192 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:18.466716 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:18.467151 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:18.467297 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:18.471587 (+ 4290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:18.471592 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:18.471595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:18.471629 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:18.474536 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:18.475583 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:18.475587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:18.475590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:18.475718 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:18.476363 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:18.476367 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:18.476369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:18.476684 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:18.476889 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:18.477655 (+ 766us) server_negotiation.cc:299] Negotiation successful
0925 05:00:18.477893 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 05:00:18.479312 28834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:18.459594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:40523 (local address 127.0.0.1:34410)
0925 05:00:18.460465 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:18.460508 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:18.462328 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:18.463085 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:18.463094 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:18.463635 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:00:18.464316 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:18.464336 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:18.465367 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:18.465371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:18.466560 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:18.466572 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:18.467384 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:18.467392 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:18.467636 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:18.468405 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:00:18.468428 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:00:18.471315 (+ 2887us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:00:18.474702 (+ 3387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:18.474714 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:18.474736 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:18.475431 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:18.475907 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:18.475916 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:18.475923 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:18.476184 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:18.476857 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:00:18.476865 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:00:18.477256 (+ 391us) client_negotiation.cc:769] Sending connection context
0925 05:00:18.477555 (+ 299us) client_negotiation.cc:240] Negotiation successful
0925 05:00:18.478128 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":414,"threads_started":1}
WARNING: no policy specified for kudu/127.6.29.188@KRBTEST.COM; defaulting to no policy
I20260925 05:00:18.481230 28832 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.29.189" port: 36751 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.188@KRBTEST.COM" created.
Sep 25 05:00:18 dist-test-slave-xzjr krb5kdc[28678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312418, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.189@KRBTEST.COM for kudu/127.6.29.189@KRBTEST.COM
I20260925 05:00:18.498107 28836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:18.482271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40848 (local address 127.6.29.189:36751)
0925 05:00:18.482890 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:18.482896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:18.482921 (+ 25us) server_negotiation.cc:407] Connection header received
0925 05:00:18.482972 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:18.482980 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:18.483171 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:18.483346 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:18.484212 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:18.486004 (+ 1792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:18.487164 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:18.488145 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:18.488344 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:18.491590 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:18.491608 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:18.491628 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:18.491674 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:18.494950 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:18.495896 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:18.495903 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:18.495906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:18.496026 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:18.496480 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:18.496484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:18.496486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:18.496832 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:18.496998 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:18.497493 (+ 495us) server_negotiation.cc:299] Negotiation successful
0925 05:00:18.497784 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":479,"thread_start_us":352,"threads_started":1}
I20260925 05:00:18.498210 28834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:18.481984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:36751 (local address 127.0.0.1:40848)
0925 05:00:18.482187 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:18.482222 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:18.482310 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:18.483425 (+ 1115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:18.483433 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:18.483877 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:00:18.484028 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:18.484037 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:18.486162 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:18.486167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:18.487012 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:18.487027 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:18.488394 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:18.488400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:18.488541 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:18.489204 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:00:18.489222 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:00:18.491361 (+ 2139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:00:18.495098 (+ 3737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:18.495107 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:18.495114 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:18.495680 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:18.496151 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:18.496159 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:18.496166 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:18.496363 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:18.496972 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:00:18.496980 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:00:18.497162 (+ 182us) client_negotiation.cc:769] Sending connection context
0925 05:00:18.497323 (+ 161us) client_negotiation.cc:240] Negotiation successful
0925 05:00:18.497656 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 05:00:18.500808 28832 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.29.188" port: 42981 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:00:18.502027 28834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:18.501424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:42981 (local address 127.0.0.1:34344)
0925 05:00:18.501640 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:18.501816 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:42981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260925 05:00:18.502343 28783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.29.188:42981: connect: Connection refused (error 111)
I20260925 05:00:18.504256 28834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:18.503811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:42981 (local address 127.0.0.1:34354)
0925 05:00:18.503998 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:18.504105 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:42981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 05:00:18.504469 28832 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.29.188:42981: Network error: Client connection negotiation failed: client connection to 127.6.29.188:42981: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.188.keytab.
Entry for principal kudu/127.6.29.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.188.keytab.
I20260925 05:00:18.519353 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.188:42981
--webserver_interface=127.6.29.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.188.keytab
--principal=kudu/127.6.29.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:36155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:40523,127.6.29.189:36751,127.6.29.188:42981
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:00:18.553819 28832 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.29.188" port: 42981 } attempt: 1
I20260925 05:00:18.555084 28834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:18.554541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:42981 (local address 127.0.0.1:34358)
0925 05:00:18.554777 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:18.554904 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:42981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260925 05:00:18.556146 28834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:18.555733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:42981 (local address 127.0.0.1:34370)
0925 05:00:18.555937 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:18.556023 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:42981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260925 05:00:18.556368 28832 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.29.188:42981: Network error: Client connection negotiation failed: client connection to 127.6.29.188:42981: connect: Connection refused (error 111)
I20260925 05:00:18.624656 28832 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.29.188" port: 42981 } attempt: 2
I20260925 05:00:18.625926 28834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:18.625380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:42981 (local address 127.0.0.1:34378)
0925 05:00:18.625615 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:18.625735 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:42981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260925 05:00:18.626945 28834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:18.626566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:42981 (local address 127.0.0.1:34394)
0925 05:00:18.626731 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:18.626812 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:42981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260925 05:00:18.627153 28832 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.29.188:42981: Network error: Client connection negotiation failed: client connection to 127.6.29.188:42981: connect: Connection refused (error 111)
I20260925 05:00:18.650331 28758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.29.189" port: 36751 } attempt: 5
Sep 25 05:00:18 dist-test-slave-xzjr krb5kdc[28678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312417, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for kudu/127.6.29.189@KRBTEST.COM
I20260925 05:00:18.665469 28836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:18.651377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40850 (local address 127.6.29.189:36751)
0925 05:00:18.651561 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:18.651566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:18.651585 (+ 19us) server_negotiation.cc:407] Connection header received
0925 05:00:18.651835 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:18.651840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:18.651908 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:18.651997 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:18.653039 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:18.653766 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:18.655479 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:18.656043 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:18.656146 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:18.660037 (+ 3891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:18.660042 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:18.660046 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:18.660080 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:18.662734 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:18.663417 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:18.663424 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:18.663427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:18.663563 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:18.664042 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:18.664048 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:18.664052 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:18.664339 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:18.664510 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:18.664922 (+ 412us) server_negotiation.cc:299] Negotiation successful
0925 05:00:18.665145 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 05:00:18.665478 28762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:18.651213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:36751 (local address 127.0.0.1:40850)
0925 05:00:18.651466 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:18.651489 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:18.651639 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:18.652071 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:18.652076 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:18.652581 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:00:18.652820 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:18.652833 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:18.653917 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:18.653922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:18.655297 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:18.655315 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:18.656221 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:18.656230 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:18.656452 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:18.657148 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:00:18.657172 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:00:18.659836 (+ 2664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:00:18.662885 (+ 3049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:18.662891 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:18.662895 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:18.663290 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:18.663682 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:18.663687 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:18.663690 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:18.663909 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:18.664464 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:00:18.664468 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:00:18.664648 (+ 180us) client_negotiation.cc:769] Sending connection context
0925 05:00:18.664803 (+ 155us) client_negotiation.cc:240] Negotiation successful
0925 05:00:18.665117 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260925 05:00:18.666579 28758 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.29.188" port: 42981 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:00:18.667618 28762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:18.667215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:42981 (local address 127.0.0.1:34408)
0925 05:00:18.667391 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:18.667475 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:42981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 05:00:18.668707 28762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:18.668342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:42981 (local address 127.0.0.1:34420)
0925 05:00:18.668540 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:18.668607 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:42981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260925 05:00:18.668943 28758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.29.188:42981: Network error: Client connection negotiation failed: client connection to 127.6.29.188:42981: connect: Connection refused (error 111)
I20260925 05:00:18.718410 28832 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.29.188" port: 42981 } attempt: 3
I20260925 05:00:18.719734 28834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:18.719161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:42981 (local address 127.0.0.1:34428)
0925 05:00:18.719408 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:18.719534 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:42981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 05:00:18.720222 28758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.29.188" port: 42981 } attempt: 1
I20260925 05:00:18.721073 28834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:18.720585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:42981 (local address 127.0.0.1:34430)
0925 05:00:18.720842 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:18.720932 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:42981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260925 05:00:18.721318 28762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:18.720834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:42981 (local address 127.0.0.1:34446)
0925 05:00:18.721005 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:18.721096 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:42981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 05:00:18.721318 28832 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.29.188:42981: Network error: Client connection negotiation failed: client connection to 127.6.29.188:42981: connect: Connection refused (error 111)
I20260925 05:00:18.722321 28762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:18.721941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:42981 (local address 127.0.0.1:34458)
0925 05:00:18.722144 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:18.722218 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:42981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260925 05:00:18.722553 28758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.29.188:42981: Network error: Client connection negotiation failed: client connection to 127.6.29.188:42981: connect: Connection refused (error 111)
I20260925 05:00:18.790798 28758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.29.188" port: 42981 } attempt: 2
I20260925 05:00:18.792181 28762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:18.791650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:42981 (local address 127.0.0.1:34470)
0925 05:00:18.791918 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:18.792019 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:42981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 05:00:18.793262 28762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:18.792913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:42981 (local address 127.0.0.1:34482)
0925 05:00:18.793083 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:18.793152 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:42981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 05:00:18.793534 28758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.29.188:42981: Network error: Client connection negotiation failed: client connection to 127.6.29.188:42981: connect: Connection refused (error 111)
W20260925 05:00:18.855379 28839 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 05:00:18.855743 28839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:00:18.855890 28839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:00:18.856019 28839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:00:18.863431 28839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:00:18.863518 28839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:00:18.863642 28839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:00:18.863723 28839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:00:18.863847 28839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:00:18.864524 28832 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.29.188" port: 42981 } attempt: 4
I20260925 05:00:18.865729 28834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:18.865220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:42981 (local address 127.0.0.1:34498)
0925 05:00:18.865473 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:18.865579 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:42981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260925 05:00:18.866798 28834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:18.866497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:42981 (local address 127.0.0.1:34500)
0925 05:00:18.866630 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:18.866700 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:42981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 05:00:18.867089 28832 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.29.188:42981: Network error: Client connection negotiation failed: client connection to 127.6.29.188:42981: connect: Connection refused (error 111)
I20260925 05:00:18.874825 28839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:36155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:40523,127.6.29.189:36751,127.6.29.188:42981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.188.keytab
--principal=kudu/127.6.29.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.188:42981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.29.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:00:18.876305 28839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:00:18.877647 28839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:00:18.889995 28845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:18.890086 28847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:18.890120 28844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:00:18.891485 28839 server_base.cc:1034] running on GCE node
I20260925 05:00:18.892409 28839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:00:18.893882 28839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:00:18.895174 28839 hybrid_clock.cc:648] HybridClock initialized: now 1790312418895110 us; error 41 us; skew 500 ppm
I20260925 05:00:18.895531 28839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:00:18 dist-test-slave-xzjr krb5kdc[28678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312418, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:00:18.898864 28839 init.cc:377] Logged in from keytab as kudu/127.6.29.188@KRBTEST.COM (short username kudu)
I20260925 05:00:18.899786 28758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.29.188" port: 42981 } attempt: 3
I20260925 05:00:18.900566 28839 webserver.cc:460] Webserver started at http://127.6.29.188:39931/ using document root <none> and password file <none>
I20260925 05:00:18.900960 28762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:18.900395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:42981 (local address 127.0.0.1:34502)
0925 05:00:18.900671 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:18.900772 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:42981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 05:00:18.901506 28839 fs_manager.cc:359] Metadata directory not provided
I20260925 05:00:18.901635 28839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:00:18.901968 28762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:18.901637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:42981 (local address 127.0.0.1:34504)
0925 05:00:18.901789 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:18.901865 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:42981: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 05:00:18.901978 28839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
W20260925 05:00:18.902202 28758 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.29.188:42981: Network error: Client connection negotiation failed: client connection to 127.6.29.188:42981: connect: Connection refused (error 111)
I20260925 05:00:18.905166 28839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-2/data/instance:
uuid: "d55cf5b1650e43a4a65d636e927cc7a6"
format_stamp: "Formatted at 2026-09-25 05:00:18 on dist-test-slave-xzjr"
server_key: "090f92fc2478c9a0f67909ba2e9f4df1"
server_key_iv: "70febe16a12cb4cb161f90bf883c941d"
server_key_version: "encryptionkey@0"
I20260925 05:00:18.905956 28839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-2/wal/instance:
uuid: "d55cf5b1650e43a4a65d636e927cc7a6"
format_stamp: "Formatted at 2026-09-25 05:00:18 on dist-test-slave-xzjr"
server_key: "090f92fc2478c9a0f67909ba2e9f4df1"
server_key_iv: "70febe16a12cb4cb161f90bf883c941d"
server_key_version: "encryptionkey@0"
I20260925 05:00:18.911180 28839 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 05:00:18.914808 28854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:18.916412 28839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 05:00:18.916591 28839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-2/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-2/wal
uuid: "d55cf5b1650e43a4a65d636e927cc7a6"
format_stamp: "Formatted at 2026-09-25 05:00:18 on dist-test-slave-xzjr"
server_key: "090f92fc2478c9a0f67909ba2e9f4df1"
server_key_iv: "70febe16a12cb4cb161f90bf883c941d"
server_key_version: "encryptionkey@0"
I20260925 05:00:18.916827 28839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:00:18.937076 28839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:00:18.941527 28839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:00:18.941845 28839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:00:18.961351 28839 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.188:42981
I20260925 05:00:18.961359 28905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.188:42981 every 8 connection(s)
I20260925 05:00:18.962688 28839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-2/data/info.pb
I20260925 05:00:18.964819 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 28839
I20260925 05:00:18.965057 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790312053666503-6262-0/minicluster-data/master-2/wal/instance
I20260925 05:00:18.965553 6262 external_mini_cluster.cc:1428] Setting key 2325b8d60e52e38adc53239004b567db
I20260925 05:00:18.966761 28906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:18.974983 28906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40523 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:00:18 dist-test-slave-xzjr krb5kdc[28678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.188@KRBTEST.COM
Sep 25 05:00:18 dist-test-slave-xzjr krb5kdc[28678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312418, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.188@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 05:00:18.988054 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:18.967383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34512 (local address 127.6.29.188:42981)
0925 05:00:18.968312 (+ 929us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:18.968328 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:18.968378 (+ 50us) server_negotiation.cc:407] Connection header received
0925 05:00:18.969988 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:18.970010 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:18.970545 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:18.970963 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:18.972532 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:18.973822 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:18.975134 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:18.975954 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:18.976150 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:18.980822 (+ 4672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:18.980829 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:18.980839 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:18.980877 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:18.983953 (+ 3076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:18.984663 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:18.984671 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:18.984686 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:18.984864 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:18.985278 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:18.985283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:18.985288 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:18.985793 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:18.986109 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:18.986569 (+ 460us) server_negotiation.cc:299] Negotiation successful
0925 05:00:18.986981 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":433,"threads_started":1}
I20260925 05:00:18.993822 28910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:18.977585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:40523 (local address 127.0.0.1:34424)
0925 05:00:18.978401 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:18.978441 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:18.978601 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:18.979002 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:18.979006 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:18.979903 (+ 897us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:00:18.980187 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:18.980200 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:18.981464 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:18.981471 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:18.982800 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:18.982815 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:18.983587 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:18.983593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:18.983840 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:18.984440 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:00:18.984457 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:00:18.986559 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:00:18.990917 (+ 4358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:18.990924 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:18.990930 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:18.991448 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:18.991981 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:18.991988 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:18.991992 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:18.992137 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:18.992737 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:00:18.992745 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:00:18.992909 (+ 164us) client_negotiation.cc:769] Sending connection context
0925 05:00:18.993081 (+ 172us) client_negotiation.cc:240] Negotiation successful
0925 05:00:18.993388 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":439,"threads_started":1}
I20260925 05:00:18.993865 28761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:18.977613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34424 (local address 127.6.29.190:40523)
0925 05:00:18.977841 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:18.977849 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:18.978557 (+ 708us) server_negotiation.cc:407] Connection header received
0925 05:00:18.978729 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:18.978734 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:18.978827 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:18.978950 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:18.980344 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:18.981238 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:18.982957 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:18.983387 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:18.983530 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:18.986766 (+ 3236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:18.986789 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:18.986797 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:18.986844 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:18.990771 (+ 3927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:18.991609 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:18.991618 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:18.991625 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:18.991861 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:18.992264 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:18.992270 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:18.992273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:18.992600 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:18.992764 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:18.993187 (+ 423us) server_negotiation.cc:299] Negotiation successful
0925 05:00:18.993388 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":68}
I20260925 05:00:18.995157 28906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.29.189" port: 36751 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:00:18 dist-test-slave-xzjr krb5kdc[28678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
Sep 25 05:00:18 dist-test-slave-xzjr krb5kdc[28678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312418, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.188@KRBTEST.COM for kudu/127.6.29.189@KRBTEST.COM
I20260925 05:00:19.006737 28911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:18.991476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34434 (local address 127.6.29.190:40523)
0925 05:00:18.992097 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:18.992101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:18.992121 (+ 20us) server_negotiation.cc:407] Connection header received
0925 05:00:18.992161 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:18.992164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:18.992213 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:18.992333 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:18.993604 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:18.994702 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:18.996228 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:18.996631 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:18.996771 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:19.000871 (+ 4100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:19.000875 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:19.000879 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:19.000910 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:19.003941 (+ 3031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:19.004731 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:19.004737 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:19.004740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:19.004867 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:19.005238 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:19.005243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:19.005247 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:19.005518 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:19.005786 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:19.006094 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 05:00:19.006318 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":513,"thread_start_us":332,"threads_started":1}
I20260925 05:00:19.008050 28910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:18.995973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:36751 (local address 127.0.0.1:40852)
0925 05:00:18.996150 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:18.996171 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:18.996262 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:18.996557 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:18.996561 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:18.996920 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:00:18.997058 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:18.997065 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:18.998017 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:18.998032 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:18.998886 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:18.998897 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:18.999612 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:18.999618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:18.999877 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:19.000455 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:00:19.000471 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:00:19.002482 (+ 2011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:00:19.005572 (+ 3090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:19.005578 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:19.005582 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:19.005928 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:19.006332 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:19.006337 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:19.006341 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:19.006541 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:19.007152 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:00:19.007156 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:00:19.007321 (+ 165us) client_negotiation.cc:769] Sending connection context
0925 05:00:19.007460 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 05:00:19.007659 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 05:00:19.008391 28836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:18.996085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40852 (local address 127.6.29.189:36751)
0925 05:00:18.996269 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:18.996274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:18.996294 (+ 20us) server_negotiation.cc:407] Connection header received
0925 05:00:18.996359 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:18.996365 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:18.996434 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:18.996569 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:18.997208 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:18.997878 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:18.999025 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:18.999445 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:18.999598 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:19.002664 (+ 3066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:19.002669 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:19.002673 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:19.002706 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:19.005436 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:19.006054 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:19.006059 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:19.006063 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:19.006224 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:19.006664 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:19.006672 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:19.006677 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:19.007028 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:19.007146 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:19.007618 (+ 472us) server_negotiation.cc:299] Negotiation successful
0925 05:00:19.007876 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 05:00:19.009405 28906 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.29.188" port: 42981 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:00:19 dist-test-slave-xzjr krb5kdc[28678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312418, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.188@KRBTEST.COM for kudu/127.6.29.188@KRBTEST.COM
Sep 25 05:00:19 dist-test-slave-xzjr krb5kdc[28678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.189@KRBTEST.COM
I20260925 05:00:19.022307 28910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:19.010077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:42981 (local address 127.0.0.1:34522)
0925 05:00:19.010294 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:19.010307 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:19.010385 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:19.010664 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:19.010667 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:19.010989 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:00:19.011172 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:19.011180 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:19.012154 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:19.012162 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:19.012977 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:19.012987 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:19.013677 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:19.013682 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:19.013917 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:19.014426 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:00:19.014446 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:00:19.016546 (+ 2100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:00:19.019658 (+ 3112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:19.019666 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:19.019670 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:19.020213 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:19.020581 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:19.020588 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:19.020593 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:19.020721 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:19.021240 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:00:19.021245 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:00:19.021387 (+ 142us) client_negotiation.cc:769] Sending connection context
0925 05:00:19.021547 (+ 160us) client_negotiation.cc:240] Negotiation successful
0925 05:00:19.021879 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":1}
I20260925 05:00:19.022332 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:19.010133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34522 (local address 127.6.29.188:42981)
0925 05:00:19.010249 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:19.010253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:19.010378 (+ 125us) server_negotiation.cc:407] Connection header received
0925 05:00:19.010462 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:19.010468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:19.010542 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:19.010639 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:19.011303 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:19.012039 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:19.013117 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:19.013519 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:19.013662 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:19.016704 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:19.016712 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:19.016721 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:19.016747 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:19.019527 (+ 2780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:19.020332 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:19.020337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:19.020342 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:19.020466 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:19.020823 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:19.020830 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:19.020833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:19.021086 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:19.021276 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:19.021632 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 05:00:19.022033 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":172,"server-negotiator.queue_time_us":29}
I20260925 05:00:19.023579 28912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:19.008579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40858 (local address 127.6.29.189:36751)
0925 05:00:19.009137 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:19.009141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:19.009159 (+ 18us) server_negotiation.cc:407] Connection header received
0925 05:00:19.009196 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:19.009200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:19.009250 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:19.009374 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:19.010902 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:19.011686 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:19.012891 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:19.013304 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:19.013437 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:19.017682 (+ 4245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:19.017687 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:19.017690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:19.017718 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:19.020524 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:19.021159 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:19.021190 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:19.021195 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:19.021350 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:19.021762 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:19.021768 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:19.021771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:19.022037 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:19.022306 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:19.023132 (+ 826us) server_negotiation.cc:299] Negotiation successful
0925 05:00:19.023299 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":260,"threads_started":1}
I20260925 05:00:19.027572 28906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d55cf5b1650e43a4a65d636e927cc7a6: Bootstrap starting.
I20260925 05:00:19.028643 6262 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:00:19.031639 28906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d55cf5b1650e43a4a65d636e927cc7a6: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:19.033056 28906 log.cc:824] T 00000000000000000000000000000000 P d55cf5b1650e43a4a65d636e927cc7a6: Log is configured to *not* fsync() on all Append() calls
I20260925 05:00:19.032271 28733 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3efa36f01dce409a897f14c46709858b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34434
I20260925 05:00:19.036409 28906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d55cf5b1650e43a4a65d636e927cc7a6: No bootstrap required, opened a new log
I20260925 05:00:19.047165 28906 raft_consensus.cc:348] T 00000000000000000000000000000000 P d55cf5b1650e43a4a65d636e927cc7a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efa36f01dce409a897f14c46709858b" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40523 } } peers { permanent_uuid: "11b833178e1e4ee583a3e21b0d502304" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 36751 } } peers { permanent_uuid: "d55cf5b1650e43a4a65d636e927cc7a6" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 42981 } }
I20260925 05:00:19.047605 28906 raft_consensus.cc:729] T 00000000000000000000000000000000 P d55cf5b1650e43a4a65d636e927cc7a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d55cf5b1650e43a4a65d636e927cc7a6, State: Initialized, Role: FOLLOWER
I20260925 05:00:19.048163 28906 consensus_queue.cc:260] T 00000000000000000000000000000000 P d55cf5b1650e43a4a65d636e927cc7a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efa36f01dce409a897f14c46709858b" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40523 } } peers { permanent_uuid: "11b833178e1e4ee583a3e21b0d502304" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 36751 } } peers { permanent_uuid: "d55cf5b1650e43a4a65d636e927cc7a6" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 42981 } }
I20260925 05:00:19.049091 28914 sys_catalog.cc:455] T 00000000000000000000000000000000 P d55cf5b1650e43a4a65d636e927cc7a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efa36f01dce409a897f14c46709858b" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40523 } } peers { permanent_uuid: "11b833178e1e4ee583a3e21b0d502304" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 36751 } } peers { permanent_uuid: "d55cf5b1650e43a4a65d636e927cc7a6" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 42981 } } }
I20260925 05:00:19.049427 28914 sys_catalog.cc:458] T 00000000000000000000000000000000 P d55cf5b1650e43a4a65d636e927cc7a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:00:19.050379 28906 sys_catalog.cc:564] T 00000000000000000000000000000000 P d55cf5b1650e43a4a65d636e927cc7a6 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 05:00:19.060451 28925 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d55cf5b1650e43a4a65d636e927cc7a6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:00:19.060585 28925 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:00:19.058099 28839 master_runner.cc:186] Error getting master registration for 127.6.29.190:40523: OK, instance_id { permanent_uuid: "3efa36f01dce409a897f14c46709858b" instance_seqno: 1790312417937057 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 05:00:19.069375 28839 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 05:00:19.079428 28758 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.29.188" port: 42981 } attempt: 4
Sep 25 05:00:19 dist-test-slave-xzjr krb5kdc[28678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312417, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for kudu/127.6.29.188@KRBTEST.COM
I20260925 05:00:19.096145 28762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:19.080191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:42981 (local address 127.0.0.1:34538)
0925 05:00:19.080435 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:19.080456 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:19.080587 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:19.081042 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:19.081047 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:19.081574 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:00:19.081794 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:19.081807 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:19.083251 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:19.083256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:19.084746 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:19.084762 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:19.085827 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:19.085837 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:19.086043 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:19.086744 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:00:19.086762 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:00:19.089119 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:00:19.093226 (+ 4107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:19.093234 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:19.093240 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:19.093773 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:19.094176 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:19.094183 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:19.094187 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:19.094344 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:19.095055 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:00:19.095059 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:00:19.095242 (+ 183us) client_negotiation.cc:769] Sending connection context
0925 05:00:19.095393 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 05:00:19.095689 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 05:00:19.096151 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:19.080473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34538 (local address 127.6.29.188:42981)
0925 05:00:19.080699 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:19.080706 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:19.080729 (+ 23us) server_negotiation.cc:407] Connection header received
0925 05:00:19.080794 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:19.080800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:19.080880 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:19.080994 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:19.081959 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:19.083115 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:19.084906 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:19.085669 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:19.085810 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:19.089318 (+ 3508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:19.089325 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:19.089330 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:19.089372 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:19.093064 (+ 3692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:19.093901 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:19.093908 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:19.093912 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:19.094058 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:19.094466 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:19.094472 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:19.094475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:19.094930 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:19.095080 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:19.095523 (+ 443us) server_negotiation.cc:299] Negotiation successful
0925 05:00:19.095743 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 05:00:19.102238 28758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3efa36f01dce409a897f14c46709858b: Bootstrap starting.
I20260925 05:00:19.106524 28758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3efa36f01dce409a897f14c46709858b: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:19.107638 28758 log.cc:824] T 00000000000000000000000000000000 P 3efa36f01dce409a897f14c46709858b: Log is configured to *not* fsync() on all Append() calls
I20260925 05:00:19.117478 28758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3efa36f01dce409a897f14c46709858b: No bootstrap required, opened a new log
I20260925 05:00:19.125582 28758 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3efa36f01dce409a897f14c46709858b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efa36f01dce409a897f14c46709858b" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40523 } } peers { permanent_uuid: "11b833178e1e4ee583a3e21b0d502304" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 36751 } } peers { permanent_uuid: "d55cf5b1650e43a4a65d636e927cc7a6" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 42981 } }
I20260925 05:00:19.126071 28758 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3efa36f01dce409a897f14c46709858b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efa36f01dce409a897f14c46709858b, State: Initialized, Role: FOLLOWER
I20260925 05:00:19.126637 28758 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3efa36f01dce409a897f14c46709858b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efa36f01dce409a897f14c46709858b" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40523 } } peers { permanent_uuid: "11b833178e1e4ee583a3e21b0d502304" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 36751 } } peers { permanent_uuid: "d55cf5b1650e43a4a65d636e927cc7a6" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 42981 } }
I20260925 05:00:19.127667 28928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3efa36f01dce409a897f14c46709858b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efa36f01dce409a897f14c46709858b" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40523 } } peers { permanent_uuid: "11b833178e1e4ee583a3e21b0d502304" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 36751 } } peers { permanent_uuid: "d55cf5b1650e43a4a65d636e927cc7a6" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 42981 } } }
I20260925 05:00:19.128089 28928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3efa36f01dce409a897f14c46709858b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:00:19.128921 28758 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3efa36f01dce409a897f14c46709858b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:00:19.129029 28733 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3efa36f01dce409a897f14c46709858b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 05:00:19.129245 28733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3efa36f01dce409a897f14c46709858b [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:19.132912 28733 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3efa36f01dce409a897f14c46709858b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efa36f01dce409a897f14c46709858b" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40523 } } peers { permanent_uuid: "11b833178e1e4ee583a3e21b0d502304" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 36751 } } peers { permanent_uuid: "d55cf5b1650e43a4a65d636e927cc7a6" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 42981 } }
I20260925 05:00:19.134501 28733 leader_election.cc:288] T 00000000000000000000000000000000 P 3efa36f01dce409a897f14c46709858b [CANDIDATE]: Term 1 election: Requested vote from peers 11b833178e1e4ee583a3e21b0d502304 (127.6.29.189:36751), d55cf5b1650e43a4a65d636e927cc7a6 (127.6.29.188:42981)
I20260925 05:00:19.136345 28881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3efa36f01dce409a897f14c46709858b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d55cf5b1650e43a4a65d636e927cc7a6"
I20260925 05:00:19.136926 28881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d55cf5b1650e43a4a65d636e927cc7a6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:19.141812 28881 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d55cf5b1650e43a4a65d636e927cc7a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efa36f01dce409a897f14c46709858b in term 1.
I20260925 05:00:19.143172 28709 leader_election.cc:302] T 00000000000000000000000000000000 P 3efa36f01dce409a897f14c46709858b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efa36f01dce409a897f14c46709858b, d55cf5b1650e43a4a65d636e927cc7a6; no voters:
I20260925 05:00:19.143813 28928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3efa36f01dce409a897f14c46709858b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:00:19.155610 28928 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3efa36f01dce409a897f14c46709858b [term 1 LEADER]: Becoming Leader. State: Replica: 3efa36f01dce409a897f14c46709858b, State: Running, Role: LEADER
I20260925 05:00:19.157262 28928 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3efa36f01dce409a897f14c46709858b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efa36f01dce409a897f14c46709858b" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40523 } } peers { permanent_uuid: "11b833178e1e4ee583a3e21b0d502304" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 36751 } } peers { permanent_uuid: "d55cf5b1650e43a4a65d636e927cc7a6" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 42981 } }
I20260925 05:00:19.166277 28832 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.29.188" port: 42981 } attempt: 5
W20260925 05:00:19.176170 28712 tablet.cc:2369] T 00000000000000000000000000000000 P 3efa36f01dce409a897f14c46709858b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 05:00:19 dist-test-slave-xzjr krb5kdc[28678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312418, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.189@KRBTEST.COM for kudu/127.6.29.188@KRBTEST.COM
I20260925 05:00:19.187232 28834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:19.167031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:42981 (local address 127.0.0.1:34544)
0925 05:00:19.167321 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:19.167342 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:19.167468 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:19.168174 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:19.168183 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:19.168787 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:00:19.169056 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:19.169073 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:19.170854 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:19.170860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:19.172424 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:19.172442 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:19.173712 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:19.173718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:19.173922 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:19.174707 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:00:19.174739 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:00:19.179187 (+ 4448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:00:19.184087 (+ 4900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:19.184093 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:19.184097 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:19.184724 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:19.185223 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:19.185228 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:19.185231 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:19.185413 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:19.186116 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:00:19.186121 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:00:19.186260 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 05:00:19.186419 (+ 159us) client_negotiation.cc:240] Negotiation successful
0925 05:00:19.186759 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167}
I20260925 05:00:19.188479 28909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:19.167390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34544 (local address 127.6.29.188:42981)
0925 05:00:19.167584 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:19.167591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:19.167611 (+ 20us) server_negotiation.cc:407] Connection header received
0925 05:00:19.167673 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:19.167678 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:19.167820 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:19.167942 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:19.169274 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:19.170678 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:19.172619 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:19.173344 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:19.173469 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:19.179447 (+ 5978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:19.179457 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:19.179465 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:19.179519 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:19.183888 (+ 4369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:19.184882 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:19.184890 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:19.184896 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:19.185099 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:19.185547 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:19.185563 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:19.185566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:19.185914 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:19.186054 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:19.187642 (+ 1588us) server_negotiation.cc:299] Negotiation successful
0925 05:00:19.187969 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 05:00:19.194098 28860 tablet.cc:2369] T 00000000000000000000000000000000 P d55cf5b1650e43a4a65d636e927cc7a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:00:19.196257 28832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11b833178e1e4ee583a3e21b0d502304: Bootstrap starting.
I20260925 05:00:19.201567 28832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11b833178e1e4ee583a3e21b0d502304: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:19.204384 28832 log.cc:824] T 00000000000000000000000000000000 P 11b833178e1e4ee583a3e21b0d502304: Log is configured to *not* fsync() on all Append() calls
I20260925 05:00:19.206686 28933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3efa36f01dce409a897f14c46709858b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3efa36f01dce409a897f14c46709858b. Latest consensus state: current_term: 1 leader_uuid: "3efa36f01dce409a897f14c46709858b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efa36f01dce409a897f14c46709858b" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40523 } } peers { permanent_uuid: "11b833178e1e4ee583a3e21b0d502304" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 36751 } } peers { permanent_uuid: "d55cf5b1650e43a4a65d636e927cc7a6" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 42981 } } }
I20260925 05:00:19.207171 28933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3efa36f01dce409a897f14c46709858b [sys.catalog]: This master's current role is: LEADER
I20260925 05:00:19.213377 28832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11b833178e1e4ee583a3e21b0d502304: No bootstrap required, opened a new log
I20260925 05:00:19.219319 28944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:00:19.233059 28940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:19.171178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34558 (local address 127.6.29.188:42981)
0925 05:00:19.186062 (+ 14884us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:19.186068 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:19.186092 (+ 24us) server_negotiation.cc:407] Connection header received
0925 05:00:19.186143 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:19.186157 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:19.186225 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:19.186325 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:19.192220 (+ 5895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:19.193504 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:19.197056 (+ 3552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:19.197731 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:19.197856 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:19.210871 (+ 13015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:19.210879 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:19.210890 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:19.210931 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:19.215434 (+ 4503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:19.222767 (+ 7333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:19.222773 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:19.222777 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:19.222952 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:19.227882 (+ 4930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:19.227889 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:19.227893 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:19.228324 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:19.228601 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:19.232350 (+ 3749us) server_negotiation.cc:299] Negotiation successful
0925 05:00:19.232613 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14740,"thread_start_us":16295,"threads_started":1}
I20260925 05:00:19.234338 28912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:19.167151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40860 (local address 127.6.29.189:36751)
0925 05:00:19.167692 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:19.167697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:19.171957 (+ 4260us) server_negotiation.cc:407] Connection header received
0925 05:00:19.172855 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:19.172861 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:19.172935 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:19.173002 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:19.185204 (+ 12202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:19.186255 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:19.203465 (+ 17210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:19.204213 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:19.204313 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:19.218071 (+ 13758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:19.218079 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:19.218084 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:19.218125 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:19.221877 (+ 3752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:19.229567 (+ 7690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:19.229576 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:19.229579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:19.229744 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:19.230503 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:19.230509 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:19.230513 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:19.230856 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:19.231088 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:19.233706 (+ 2618us) server_negotiation.cc:299] Negotiation successful
0925 05:00:19.233963 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414}
I20260925 05:00:19.237325 28911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:19.170426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34444 (local address 127.6.29.190:40523)
0925 05:00:19.170772 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:19.170778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:19.189702 (+ 18924us) server_negotiation.cc:407] Connection header received
0925 05:00:19.190106 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:19.190112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:19.190212 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:19.190282 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:19.198126 (+ 7844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:19.199226 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:19.205306 (+ 6080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:19.206095 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:19.206215 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:19.223488 (+ 17273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:19.223495 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:19.223502 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:19.223546 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:19.227251 (+ 3705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:19.231545 (+ 4294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:19.231551 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:19.231555 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:19.231750 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:19.235334 (+ 3584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:19.235339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:19.235343 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:19.235814 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:19.236059 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:19.236681 (+ 622us) server_negotiation.cc:299] Negotiation successful
0925 05:00:19.236933 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 05:00:19.237895 28832 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11b833178e1e4ee583a3e21b0d502304 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efa36f01dce409a897f14c46709858b" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40523 } } peers { permanent_uuid: "11b833178e1e4ee583a3e21b0d502304" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 36751 } } peers { permanent_uuid: "d55cf5b1650e43a4a65d636e927cc7a6" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 42981 } }
I20260925 05:00:19.244139 28832 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11b833178e1e4ee583a3e21b0d502304 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11b833178e1e4ee583a3e21b0d502304, State: Initialized, Role: FOLLOWER
I20260925 05:00:19.245025 28832 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11b833178e1e4ee583a3e21b0d502304 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efa36f01dce409a897f14c46709858b" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40523 } } peers { permanent_uuid: "11b833178e1e4ee583a3e21b0d502304" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 36751 } } peers { permanent_uuid: "d55cf5b1650e43a4a65d636e927cc7a6" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 42981 } }
I20260925 05:00:19.248420 28944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:00:19.249977 28952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11b833178e1e4ee583a3e21b0d502304 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efa36f01dce409a897f14c46709858b" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40523 } } peers { permanent_uuid: "11b833178e1e4ee583a3e21b0d502304" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 36751 } } peers { permanent_uuid: "d55cf5b1650e43a4a65d636e927cc7a6" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 42981 } } }
I20260925 05:00:19.252290 28832 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11b833178e1e4ee583a3e21b0d502304 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:00:19.252699 28807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3efa36f01dce409a897f14c46709858b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "11b833178e1e4ee583a3e21b0d502304"
I20260925 05:00:19.254908 28807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11b833178e1e4ee583a3e21b0d502304 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:19.258404 28807 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 11b833178e1e4ee583a3e21b0d502304 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efa36f01dce409a897f14c46709858b in term 1.
I20260925 05:00:19.253283 28952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11b833178e1e4ee583a3e21b0d502304 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:00:19.262557 28881 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d55cf5b1650e43a4a65d636e927cc7a6 [term 1 FOLLOWER]: Refusing update from remote peer 3efa36f01dce409a897f14c46709858b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:00:19.264039 28928 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3efa36f01dce409a897f14c46709858b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d55cf5b1650e43a4a65d636e927cc7a6" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 42981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:00:19.264678 28807 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 11b833178e1e4ee583a3e21b0d502304 [term 1 FOLLOWER]: Refusing update from remote peer 3efa36f01dce409a897f14c46709858b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:00:19.268190 28928 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3efa36f01dce409a897f14c46709858b [LEADER]: Connected to new peer: Peer: permanent_uuid: "11b833178e1e4ee583a3e21b0d502304" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 36751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:00:19.268664 28691 master_runner.cc:186] Error getting master registration for 127.6.29.189:36751: OK, instance_id { permanent_uuid: "11b833178e1e4ee583a3e21b0d502304" instance_seqno: 1790312418438555 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 05:00:19.280409 28691 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 05:00:19.288738 28914 sys_catalog.cc:455] T 00000000000000000000000000000000 P d55cf5b1650e43a4a65d636e927cc7a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3efa36f01dce409a897f14c46709858b. Latest consensus state: current_term: 1 leader_uuid: "3efa36f01dce409a897f14c46709858b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efa36f01dce409a897f14c46709858b" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40523 } } peers { permanent_uuid: "11b833178e1e4ee583a3e21b0d502304" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 36751 } } peers { permanent_uuid: "d55cf5b1650e43a4a65d636e927cc7a6" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 42981 } } }
I20260925 05:00:19.289104 28914 sys_catalog.cc:458] T 00000000000000000000000000000000 P d55cf5b1650e43a4a65d636e927cc7a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:00:19.289173 28928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3efa36f01dce409a897f14c46709858b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3efa36f01dce409a897f14c46709858b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efa36f01dce409a897f14c46709858b" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40523 } } peers { permanent_uuid: "11b833178e1e4ee583a3e21b0d502304" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 36751 } } peers { permanent_uuid: "d55cf5b1650e43a4a65d636e927cc7a6" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 42981 } } }
I20260925 05:00:19.289530 28928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3efa36f01dce409a897f14c46709858b [sys.catalog]: This master's current role is: LEADER
I20260925 05:00:19.290987 28952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11b833178e1e4ee583a3e21b0d502304 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3efa36f01dce409a897f14c46709858b. Latest consensus state: current_term: 1 leader_uuid: "3efa36f01dce409a897f14c46709858b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efa36f01dce409a897f14c46709858b" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40523 } } peers { permanent_uuid: "11b833178e1e4ee583a3e21b0d502304" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 36751 } } peers { permanent_uuid: "d55cf5b1650e43a4a65d636e927cc7a6" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 42981 } } }
I20260925 05:00:19.291285 28952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11b833178e1e4ee583a3e21b0d502304 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:00:19.292567 28933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3efa36f01dce409a897f14c46709858b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3efa36f01dce409a897f14c46709858b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efa36f01dce409a897f14c46709858b" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40523 } } peers { permanent_uuid: "11b833178e1e4ee583a3e21b0d502304" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 36751 } } peers { permanent_uuid: "d55cf5b1650e43a4a65d636e927cc7a6" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 42981 } } }
I20260925 05:00:19.292866 28933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3efa36f01dce409a897f14c46709858b [sys.catalog]: This master's current role is: LEADER
I20260925 05:00:19.302753 28914 sys_catalog.cc:455] T 00000000000000000000000000000000 P d55cf5b1650e43a4a65d636e927cc7a6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3efa36f01dce409a897f14c46709858b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efa36f01dce409a897f14c46709858b" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40523 } } peers { permanent_uuid: "11b833178e1e4ee583a3e21b0d502304" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 36751 } } peers { permanent_uuid: "d55cf5b1650e43a4a65d636e927cc7a6" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 42981 } } }
I20260925 05:00:19.303035 28914 sys_catalog.cc:458] T 00000000000000000000000000000000 P d55cf5b1650e43a4a65d636e927cc7a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:00:19.308277 28952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11b833178e1e4ee583a3e21b0d502304 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3efa36f01dce409a897f14c46709858b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efa36f01dce409a897f14c46709858b" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 40523 } } peers { permanent_uuid: "11b833178e1e4ee583a3e21b0d502304" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 36751 } } peers { permanent_uuid: "d55cf5b1650e43a4a65d636e927cc7a6" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 42981 } } }
I20260925 05:00:19.308579 28952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11b833178e1e4ee583a3e21b0d502304 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:00:19.309119 28944 catalog_manager.cc:1347] Generated new cluster ID: aac41dd1d5ca40308f5b68500fb49f77
I20260925 05:00:19.309232 28944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:00:19.333331 28973 catalog_manager.cc:1259] Loaded cluster ID: aac41dd1d5ca40308f5b68500fb49f77
I20260925 05:00:19.333545 28973 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 11b833178e1e4ee583a3e21b0d502304: loading cluster ID for follower catalog manager: success
W20260925 05:00:19.334856 28973 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 11b833178e1e4ee583a3e21b0d502304: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260925 05:00:19.339051 28973 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:00:19.340420 28944 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:00:19.342290 28944 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:00:19.361478 28944 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3efa36f01dce409a897f14c46709858b: Generated new TSK 0
I20260925 05:00:19.362802 28944 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 05:00:19.475400 28911 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:00:19.466721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41102 (local address 127.6.29.190:40523)
0925 05:00:19.466933 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:19.466943 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:19.472875 (+ 5932us) server_negotiation.cc:407] Connection header received
0925 05:00:19.473035 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:19.473040 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:19.473124 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:19.473238 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:19.475123 (+ 1885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41102: BlockingRecv error: recv got EOF from 127.0.0.1:41102 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260925 05:00:19.479171 28940 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:00:19.473509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55350 (local address 127.6.29.188:42981)
0925 05:00:19.473689 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:19.473695 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:19.476813 (+ 3118us) server_negotiation.cc:407] Connection header received
0925 05:00:19.477721 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:19.477729 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:19.477820 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:19.477895 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:19.478907 (+ 1012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55350: BlockingRecv error: recv got EOF from 127.0.0.1:55350 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260925 05:00:19.481432 28912 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:00:19.463025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45068 (local address 127.6.29.189:36751)
0925 05:00:19.463227 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:19.463235 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:19.472223 (+ 8988us) server_negotiation.cc:407] Connection header received
0925 05:00:19.472445 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:19.472454 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:19.472577 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:19.472690 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:19.481068 (+ 8378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45068: BlockingRecv error: recv got EOF from 127.0.0.1:45068 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260925 05:00:19.524973 28912 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:00:19.514494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45084 (local address 127.6.29.189:36751)
0925 05:00:19.514717 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:19.514724 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:19.522580 (+ 7856us) server_negotiation.cc:407] Connection header received
0925 05:00:19.523202 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:19.523211 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:19.523385 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:19.523477 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:19.524721 (+ 1244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45084: BlockingRecv error: recv got EOF from 127.0.0.1:45084 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260925 05:00:19.574417 28912 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:00:19.564643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45090 (local address 127.6.29.189:36751)
0925 05:00:19.564887 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:19.564894 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:19.572330 (+ 7436us) server_negotiation.cc:407] Connection header received
0925 05:00:19.572861 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:19.572869 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:19.572972 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:19.573055 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:19.574129 (+ 1074us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45090: BlockingRecv error: recv got EOF from 127.0.0.1:45090 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260925 05:00:19.650588 28912 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:00:19.633469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45102 (local address 127.6.29.189:36751)
0925 05:00:19.633716 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:19.633726 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:19.648440 (+ 14714us) server_negotiation.cc:407] Connection header received
0925 05:00:19.648695 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:19.648705 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:19.648806 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:19.648926 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:19.650297 (+ 1371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45102: BlockingRecv error: recv got EOF from 127.0.0.1:45102 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260925 05:00:19.735363 28912 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:00:19.719441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45106 (local address 127.6.29.189:36751)
0925 05:00:19.719628 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:19.719634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:19.733493 (+ 13859us) server_negotiation.cc:407] Connection header received
0925 05:00:19.733752 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:19.733759 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:19.733885 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:19.733991 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:19.735105 (+ 1114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45106: BlockingRecv error: recv got EOF from 127.0.0.1:45106 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260925 05:00:19.749634 28912 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:00:19.747405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45122 (local address 127.6.29.189:36751)
0925 05:00:19.747644 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:19.747653 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:19.748213 (+ 560us) server_negotiation.cc:407] Connection header received
0925 05:00:19.748446 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:19.748450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:19.748550 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:19.748667 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:19.749382 (+ 715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45122: BlockingRecv error: recv got EOF from 127.0.0.1:45122 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260925 05:00:19.772037 28912 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:00:19.769409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45136 (local address 127.6.29.189:36751)
0925 05:00:19.769709 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:19.769721 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:19.770361 (+ 640us) server_negotiation.cc:407] Connection header received
0925 05:00:19.770623 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:19.770637 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:19.770716 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:19.770839 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:19.771682 (+ 843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45136: BlockingRecv error: recv got EOF from 127.0.0.1:45136 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
W20260925 05:00:19.810645 28912 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:00:19.808053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45152 (local address 127.6.29.189:36751)
0925 05:00:19.808347 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:19.808358 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:19.808987 (+ 629us) server_negotiation.cc:407] Connection header received
0925 05:00:19.809332 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:19.809337 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:19.809419 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:19.809557 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:19.810360 (+ 803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45152: BlockingRecv error: recv got EOF from 127.0.0.1:45152 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260925 05:00:19.890384 28912 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:00:19.887594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45154 (local address 127.6.29.189:36751)
0925 05:00:19.887926 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:19.887936 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:19.888607 (+ 671us) server_negotiation.cc:407] Connection header received
0925 05:00:19.888863 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:19.888871 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:19.889002 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:19.889150 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:19.890093 (+ 943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45154: BlockingRecv error: recv got EOF from 127.0.0.1:45154 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260925 05:00:20.023989 28912 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:00:20.021886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45164 (local address 127.6.29.189:36751)
0925 05:00:20.022139 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:20.022149 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:20.022513 (+ 364us) server_negotiation.cc:407] Connection header received
0925 05:00:20.022643 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:20.022648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:20.022725 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:20.022819 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:20.023721 (+ 902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45164: BlockingRecv error: recv got EOF from 127.0.0.1:45164 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 05:00:20.062562 28925 catalog_manager.cc:1259] Loaded cluster ID: aac41dd1d5ca40308f5b68500fb49f77
I20260925 05:00:20.062665 28925 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d55cf5b1650e43a4a65d636e927cc7a6: loading cluster ID for follower catalog manager: success
I20260925 05:00:20.065225 28925 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d55cf5b1650e43a4a65d636e927cc7a6: acquiring CA information for follower catalog manager: success
I20260925 05:00:20.066987 28925 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d55cf5b1650e43a4a65d636e927cc7a6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 05:00:20.287413 28912 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:00:20.285560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45180 (local address 127.6.29.189:36751)
0925 05:00:20.285741 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:20.285746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:20.286084 (+ 338us) server_negotiation.cc:407] Connection header received
0925 05:00:20.286227 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:20.286231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:20.286308 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:20.286406 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:20.287134 (+ 728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45180: BlockingRecv error: recv got EOF from 127.0.0.1:45180 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 05:00:20.342665 28973 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 11b833178e1e4ee583a3e21b0d502304: acquiring CA information for follower catalog manager: success
I20260925 05:00:20.344715 28973 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 11b833178e1e4ee583a3e21b0d502304: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 05:00:20.809842 29039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:20.804311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45182 (local address 127.6.29.189:36751)
0925 05:00:20.804769 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:20.804774 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:20.804824 (+ 50us) server_negotiation.cc:407] Connection header received
0925 05:00:20.804996 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:20.805001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:20.805175 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:20.805283 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:00:20.805736 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:20.806465 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:20.807754 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:20.808200 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:20.808288 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:20.808751 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:00:20.809303 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:00:20.809419 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:20.809520 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 05:00:20.809589 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":253,"threads_started":1}
I20260925 05:00:20.822548 29045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:20.816246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55354 (local address 127.6.29.188:42981)
0925 05:00:20.816964 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:20.816970 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:20.816988 (+ 18us) server_negotiation.cc:407] Connection header received
0925 05:00:20.817248 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:20.817253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:20.817416 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:20.817588 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:00:20.818009 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:20.818744 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:20.820168 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:20.820659 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:20.820764 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:20.821265 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:00:20.821886 (+ 621us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:00:20.822007 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:20.822090 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 05:00:20.822161 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":527,"threads_started":1}
I20260925 05:00:20.837227 29051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:20.829265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41106 (local address 127.6.29.190:40523)
0925 05:00:20.829850 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:20.829860 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:20.831016 (+ 1156us) server_negotiation.cc:407] Connection header received
0925 05:00:20.831173 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:20.831181 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:20.831422 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:20.831617 (+ 195us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:00:20.832236 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:20.833187 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:20.834918 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:20.835347 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:20.835482 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:20.836006 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:00:20.836402 (+ 396us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:00:20.836515 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:20.836623 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 05:00:20.836879 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":648,"threads_started":1}
I20260925 05:00:20.838456 29045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:20.830911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55362 (local address 127.6.29.188:42981)
0925 05:00:20.831111 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:20.831117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:20.831629 (+ 512us) server_negotiation.cc:407] Connection header received
0925 05:00:20.831934 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:20.831938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:20.832005 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:20.832103 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:00:20.832758 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:20.833682 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:20.836347 (+ 2665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:20.836972 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:20.837088 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:20.837602 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:00:20.837745 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:00:20.837816 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:20.838060 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 05:00:20.838145 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 05:00:20.839187 29039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:20.829598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45198 (local address 127.6.29.189:36751)
0925 05:00:20.829893 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:20.829903 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:20.831346 (+ 1443us) server_negotiation.cc:407] Connection header received
0925 05:00:20.831609 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:20.831616 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:20.831711 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:20.831852 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:00:20.832864 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:20.833850 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:20.837196 (+ 3346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:20.837737 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:20.837880 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:20.838485 (+ 605us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:00:20.838627 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:00:20.838733 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:20.838858 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 05:00:20.838946 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260925 05:00:20.851213 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 28691
I20260925 05:00:20.852404 28753 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:00:20.968662 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 28691
I20260925 05:00:20.987109 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 28765
I20260925 05:00:20.992761 28827 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:00:21.101660 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 28765
I20260925 05:00:21.122150 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 28839
I20260925 05:00:21.123426 28901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:00:21.233315 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 28839
2026-09-25T05:00:21Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5141 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 05:00:21 dist-test-slave-xzjr krb5kdc[29061](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 05:00:21 dist-test-slave-xzjr krb5kdc[29061](info): set up 2 sockets
Sep 25 05:00:21 dist-test-slave-xzjr krb5kdc[29061](info): commencing operation
krb5kdc: starting...
W20260925 05:00:22.357015 6262 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/krb5kdc/test-user.keytab.
Sep 25 05:00:22 dist-test-slave-xzjr krb5kdc[29061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T05:00:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T05:00:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.29.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
Entry for principal kudu/127.6.29.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab.
I20260925 05:00:22.627038 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:43911
--webserver_interface=127.6.29.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:40773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:43911,127.6.29.189:41607,127.6.29.188:40077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 05:00:22.968113 29074 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 05:00:22.968492 29074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:00:22.968606 29074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:00:22.968750 29074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:00:22.976275 29074 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:00:22.976369 29074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:00:22.976469 29074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:00:22.976560 29074 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:00:22.976679 29074 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:00:22.986824 29074 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:40773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:43911,127.6.29.189:41607,127.6.29.188:40077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.190.keytab
--principal=kudu/127.6.29.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.190:43911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.29.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:00:22.988250 29074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:00:22.989626 29074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:00:23.001621 29080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:23.001711 29082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:23.001672 29079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:00:23.002573 29074 server_base.cc:1034] running on GCE node
I20260925 05:00:23.003372 29074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:00:23.004940 29074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:00:23.006207 29074 hybrid_clock.cc:648] HybridClock initialized: now 1790312423006133 us; error 46 us; skew 500 ppm
I20260925 05:00:23.006556 29074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:00:23 dist-test-slave-xzjr krb5kdc[29061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312423, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:00:23.009900 29074 init.cc:377] Logged in from keytab as kudu/127.6.29.190@KRBTEST.COM (short username kudu)
I20260925 05:00:23.011370 29074 webserver.cc:460] Webserver started at http://127.6.29.190:37377/ using document root <none> and password file <none>
I20260925 05:00:23.012213 29074 fs_manager.cc:359] Metadata directory not provided
I20260925 05:00:23.012347 29074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:00:23.012676 29074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:00:23.015734 29074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-0/data/instance:
uuid: "75b663e2c09043658a3ace1ca5558b51"
format_stamp: "Formatted at 2026-09-25 05:00:23 on dist-test-slave-xzjr"
server_key: "5d258c56229436514981b9824a6c19d7"
server_key_iv: "165a065431b8f29bcb2a8b799e3addea"
server_key_version: "encryptionkey@0"
I20260925 05:00:23.016619 29074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-0/wal/instance:
uuid: "75b663e2c09043658a3ace1ca5558b51"
format_stamp: "Formatted at 2026-09-25 05:00:23 on dist-test-slave-xzjr"
server_key: "5d258c56229436514981b9824a6c19d7"
server_key_iv: "165a065431b8f29bcb2a8b799e3addea"
server_key_version: "encryptionkey@0"
I20260925 05:00:23.022156 29074 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 05:00:23.025894 29089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:23.027347 29074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:00:23.027546 29074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-0/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-0/wal
uuid: "75b663e2c09043658a3ace1ca5558b51"
format_stamp: "Formatted at 2026-09-25 05:00:23 on dist-test-slave-xzjr"
server_key: "5d258c56229436514981b9824a6c19d7"
server_key_iv: "165a065431b8f29bcb2a8b799e3addea"
server_key_version: "encryptionkey@0"
I20260925 05:00:23.027822 29074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:00:23.058552 29074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:00:23.063342 29074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:00:23.063655 29074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:00:23.082373 29074 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.190:43911
I20260925 05:00:23.082387 29140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.190:43911 every 8 connection(s)
I20260925 05:00:23.083909 29074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-0/data/info.pb
I20260925 05:00:23.087693 29141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:23.093703 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 29074
I20260925 05:00:23.093923 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-0/wal/instance
I20260925 05:00:23.094398 6262 external_mini_cluster.cc:1428] Setting key 770fa67c08be1c7b63ab93a8604633fd
I20260925 05:00:23.094527 29141 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43911 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:00:23 dist-test-slave-xzjr krb5kdc[29061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312423, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 05:00:23.120500 29143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:23.097873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48414 (local address 127.6.29.190:43911)
0925 05:00:23.098869 (+ 996us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:23.098886 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:23.099488 (+ 602us) server_negotiation.cc:407] Connection header received
0925 05:00:23.101589 (+ 2101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:23.101601 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:23.101933 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:23.102167 (+ 234us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:23.103847 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:23.105089 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:23.107344 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:23.108007 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:23.108174 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:23.112736 (+ 4562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:23.112751 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:23.112765 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:23.112799 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:23.115698 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:23.116721 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:23.116728 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:23.116730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:23.116878 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:23.117438 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:23.117442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:23.117444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:23.118042 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:23.118211 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:23.118986 (+ 775us) server_negotiation.cc:299] Negotiation successful
0925 05:00:23.119284 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":724,"thread_start_us":455,"threads_started":1}
I20260925 05:00:23.120558 29144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:23.097873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43911 (local address 127.0.0.1:48414)
0925 05:00:23.099284 (+ 1411us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:23.099333 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:23.101192 (+ 1859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:23.102182 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:23.102187 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:23.103026 (+ 839us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:00:23.103543 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:23.103569 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:23.105254 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:23.105280 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:23.107189 (+ 1909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:23.107207 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:23.108270 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:23.108281 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:23.108518 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:23.109272 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:00:23.109306 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:00:23.112438 (+ 3132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:00:23.115931 (+ 3493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:23.115947 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:23.115963 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:23.116578 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:23.117010 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:23.117018 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:23.117022 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:23.117268 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:23.118258 (+ 990us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:00:23.118266 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:00:23.118532 (+ 266us) client_negotiation.cc:769] Sending connection context
0925 05:00:23.118845 (+ 313us) client_negotiation.cc:240] Negotiation successful
0925 05:00:23.119451 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1149,"mutex_wait_us":60,"spinlock_wait_cycles":25984,"thread_start_us":874,"threads_started":1}
WARNING: no policy specified for kudu/127.6.29.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.189@KRBTEST.COM" created.
I20260925 05:00:23.123720 29141 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.29.189" port: 41607 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:00:23.125075 29144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:23.124605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:41607 (local address 127.0.0.1:33912)
0925 05:00:23.124799 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:23.124938 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.189:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260925 05:00:23.125422 29090 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.29.189:41607: connect: Connection refused (error 111)
I20260925 05:00:23.127228 29144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:23.126835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:41607 (local address 127.0.0.1:33920)
0925 05:00:23.127001 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:23.127087 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.189:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 05:00:23.127472 29141 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.29.189:41607: Network error: Client connection negotiation failed: client connection to 127.6.29.189:41607: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.189.keytab.
Entry for principal kudu/127.6.29.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.189.keytab.
I20260925 05:00:23.155483 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.189:41607
--webserver_interface=127.6.29.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.189.keytab
--principal=kudu/127.6.29.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:40773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:43911,127.6.29.189:41607,127.6.29.188:40077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:00:23.176807 29141 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.29.189" port: 41607 } attempt: 1
I20260925 05:00:23.178225 29144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:23.177636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:41607 (local address 127.0.0.1:33932)
0925 05:00:23.177877 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:23.178058 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.189:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260925 05:00:23.179445 29144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:23.179071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:41607 (local address 127.0.0.1:33948)
0925 05:00:23.179235 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:23.179308 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.189:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260925 05:00:23.179649 29141 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.29.189:41607: Network error: Client connection negotiation failed: client connection to 127.6.29.189:41607: connect: Connection refused (error 111)
I20260925 05:00:23.248569 29141 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.29.189" port: 41607 } attempt: 2
I20260925 05:00:23.249747 29144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:23.249240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:41607 (local address 127.0.0.1:33950)
0925 05:00:23.249511 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:23.249610 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.189:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260925 05:00:23.250834 29144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:23.250448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:41607 (local address 127.0.0.1:33962)
0925 05:00:23.250623 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:23.250698 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.189:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 05:00:23.251089 29141 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.29.189:41607: Network error: Client connection negotiation failed: client connection to 127.6.29.189:41607: connect: Connection refused (error 111)
I20260925 05:00:23.342375 29141 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.29.189" port: 41607 } attempt: 3
I20260925 05:00:23.343540 29144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:23.343082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:41607 (local address 127.0.0.1:33968)
0925 05:00:23.343299 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:23.343395 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.189:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260925 05:00:23.344614 29144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:23.344259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:41607 (local address 127.0.0.1:33982)
0925 05:00:23.344409 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:23.344484 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.189:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 05:00:23.344785 29141 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.29.189:41607: Network error: Client connection negotiation failed: client connection to 127.6.29.189:41607: connect: Connection refused (error 111)
I20260925 05:00:23.488013 29141 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.29.189" port: 41607 } attempt: 4
I20260925 05:00:23.489229 29144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:23.488668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:41607 (local address 127.0.0.1:33990)
0925 05:00:23.488991 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:23.489099 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.189:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260925 05:00:23.490258 29144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:23.489913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:41607 (local address 127.0.0.1:33998)
0925 05:00:23.490061 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:23.490144 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.189:41607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 05:00:23.490445 29141 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.29.189:41607: Network error: Client connection negotiation failed: client connection to 127.6.29.189:41607: connect: Connection refused (error 111)
W20260925 05:00:23.492645 29148 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 05:00:23.493013 29148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:00:23.493142 29148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:00:23.493286 29148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 05:00:23.500662 29148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:00:23.500803 29148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:00:23.500906 29148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 05:00:23.501003 29148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:00:23.501096 29148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:00:23.511155 29148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:40773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:43911,127.6.29.189:41607,127.6.29.188:40077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.189.keytab
--principal=kudu/127.6.29.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.189:41607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.29.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:00:23.512571 29148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:00:23.513970 29148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 05:00:23.526386 29154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:23.526357 29153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:23.526297 29156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:00:23.527446 29148 server_base.cc:1034] running on GCE node
I20260925 05:00:23.528400 29148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:00:23.529914 29148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:00:23.531137 29148 hybrid_clock.cc:648] HybridClock initialized: now 1790312423531048 us; error 52 us; skew 500 ppm
I20260925 05:00:23.531534 29148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:00:23 dist-test-slave-xzjr krb5kdc[29061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312423, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:00:23.534970 29148 init.cc:377] Logged in from keytab as kudu/127.6.29.189@KRBTEST.COM (short username kudu)
I20260925 05:00:23.536538 29148 webserver.cc:460] Webserver started at http://127.6.29.189:41785/ using document root <none> and password file <none>
I20260925 05:00:23.537262 29148 fs_manager.cc:359] Metadata directory not provided
I20260925 05:00:23.537395 29148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:00:23.537729 29148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:00:23.540854 29148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-1/data/instance:
uuid: "fba2a365442d40c7b44cf58d6a639540"
format_stamp: "Formatted at 2026-09-25 05:00:23 on dist-test-slave-xzjr"
server_key: "1b8b3542af72289c65af7324e5c04524"
server_key_iv: "fdfa2d969c3bf1ac10ed243cf0c58ef3"
server_key_version: "encryptionkey@0"
I20260925 05:00:23.541652 29148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-1/wal/instance:
uuid: "fba2a365442d40c7b44cf58d6a639540"
format_stamp: "Formatted at 2026-09-25 05:00:23 on dist-test-slave-xzjr"
server_key: "1b8b3542af72289c65af7324e5c04524"
server_key_iv: "fdfa2d969c3bf1ac10ed243cf0c58ef3"
server_key_version: "encryptionkey@0"
I20260925 05:00:23.546799 29148 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 05:00:23.550345 29163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:23.551620 29148 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 05:00:23.551865 29148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-1/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-1/wal
uuid: "fba2a365442d40c7b44cf58d6a639540"
format_stamp: "Formatted at 2026-09-25 05:00:23 on dist-test-slave-xzjr"
server_key: "1b8b3542af72289c65af7324e5c04524"
server_key_iv: "fdfa2d969c3bf1ac10ed243cf0c58ef3"
server_key_version: "encryptionkey@0"
I20260925 05:00:23.552086 29148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:00:23.568387 29148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:00:23.573035 29148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:00:23.573347 29148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:00:23.593108 29148 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.189:41607
I20260925 05:00:23.593147 29214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.189:41607 every 8 connection(s)
I20260925 05:00:23.594551 29148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-1/data/info.pb
I20260925 05:00:23.598378 29215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:23.601370 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 29148
I20260925 05:00:23.601581 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-1/wal/instance
I20260925 05:00:23.602105 6262 external_mini_cluster.cc:1428] Setting key 31a11f68855802b64f85590ecfea6f0e
I20260925 05:00:23.607102 29215 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43911 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:00:23 dist-test-slave-xzjr krb5kdc[29061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312423, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.189@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.29.188@KRBTEST.COM; defaulting to no policy
I20260925 05:00:23.631315 29143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:23.610731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48420 (local address 127.6.29.190:43911)
0925 05:00:23.610978 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:23.610987 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:23.612158 (+ 1171us) server_negotiation.cc:407] Connection header received
0925 05:00:23.614642 (+ 2484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:23.614647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:23.614720 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:23.614816 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:23.616933 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:23.617625 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:23.620019 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:23.620433 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:23.620527 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:23.625018 (+ 4491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:23.625024 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:23.625027 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:23.625061 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:23.627715 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:23.628749 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:23.628753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:23.628755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:23.628901 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:23.629400 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:23.629404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:23.629406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:23.629682 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:23.629801 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:23.630771 (+ 970us) server_negotiation.cc:299] Negotiation successful
0925 05:00:23.631040 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 05:00:23.632570 29217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:23.611122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43911 (local address 127.0.0.1:48420)
0925 05:00:23.611986 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:23.612041 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:23.614334 (+ 2293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:23.615078 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:23.615092 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:23.615947 (+ 855us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:00:23.616748 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:23.616774 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:23.617792 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:23.617800 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:23.619877 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:23.619897 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:23.620660 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:23.620668 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:23.620988 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:23.621768 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:00:23.621802 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:00:23.624780 (+ 2978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:00:23.627906 (+ 3126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:23.627920 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:23.627941 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:23.628625 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:23.629012 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:23.629018 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:23.629023 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:23.629282 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:23.629822 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:00:23.629832 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:00:23.630357 (+ 525us) client_negotiation.cc:769] Sending connection context
0925 05:00:23.630693 (+ 336us) client_negotiation.cc:240] Negotiation successful
0925 05:00:23.631307 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":368,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.29.188@KRBTEST.COM" created.
I20260925 05:00:23.634457 29215 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.29.189" port: 41607 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:00:23 dist-test-slave-xzjr krb5kdc[29061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312423, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.189@KRBTEST.COM for kudu/127.6.29.189@KRBTEST.COM
I20260925 05:00:23.651500 29217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:23.635292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:41607 (local address 127.0.0.1:34010)
0925 05:00:23.635542 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:23.635564 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:23.635702 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:23.636641 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:23.636652 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:23.637153 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:00:23.637366 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:23.637381 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:23.638755 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:23.638763 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:23.640278 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:23.640296 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:23.641100 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:23.641108 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:23.641350 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:23.641981 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:00:23.642019 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:00:23.644705 (+ 2686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:00:23.649136 (+ 4431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:23.649140 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:23.649142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:23.649523 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:23.649966 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:23.649970 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:23.649972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:23.650090 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:23.650806 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:00:23.650813 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:00:23.650939 (+ 126us) client_negotiation.cc:769] Sending connection context
0925 05:00:23.651039 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 05:00:23.651238 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 05:00:23.652036 29219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:23.635520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34010 (local address 127.6.29.189:41607)
0925 05:00:23.636224 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:23.636230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:23.636255 (+ 25us) server_negotiation.cc:407] Connection header received
0925 05:00:23.636301 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:23.636306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:23.636477 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:23.636599 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:23.637539 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:23.638597 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:23.640412 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:23.640946 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:23.641039 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:23.644935 (+ 3896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:23.644954 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:23.644976 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:23.645026 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:23.649001 (+ 3975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:23.649678 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:23.649690 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:23.649696 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:23.649845 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:23.650198 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:23.650206 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:23.650212 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:23.650689 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:23.650903 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:23.651246 (+ 343us) server_negotiation.cc:299] Negotiation successful
0925 05:00:23.651529 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":411,"threads_started":1}
I20260925 05:00:23.654382 29215 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.29.188" port: 40077 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:00:23.655537 29217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:23.655047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:40077 (local address 127.0.0.1:47480)
0925 05:00:23.655238 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:23.655391 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:40077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260925 05:00:23.655877 29166 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.29.188:40077: connect: Connection refused (error 111)
I20260925 05:00:23.657595 29217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:23.657276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:40077 (local address 127.0.0.1:47492)
0925 05:00:23.657417 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:23.657497 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:40077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 05:00:23.657855 29215 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.29.188:40077: Network error: Client connection negotiation failed: client connection to 127.6.29.188:40077: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.29.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.188.keytab.
Entry for principal kudu/127.6.29.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.188.keytab.
I20260925 05:00:23.671262 6262 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
/tmp/dist-test-taskjC0Yza/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.188:40077
--webserver_interface=127.6.29.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.188.keytab
--principal=kudu/127.6.29.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.29.148:40773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.29.190:43911,127.6.29.189:41607,127.6.29.188:40077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 05:00:23.707218 29215 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.29.188" port: 40077 } attempt: 1
I20260925 05:00:23.708454 29217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:23.707998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:40077 (local address 127.0.0.1:47502)
0925 05:00:23.708193 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:23.708299 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:40077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 05:00:23.709537 29217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:23.709227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:40077 (local address 127.0.0.1:47510)
0925 05:00:23.709354 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:23.709437 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:40077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 05:00:23.709823 29215 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.29.188:40077: Network error: Client connection negotiation failed: client connection to 127.6.29.188:40077: connect: Connection refused (error 111)
I20260925 05:00:23.778079 29215 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.29.188" port: 40077 } attempt: 2
I20260925 05:00:23.779335 29217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:23.778876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:40077 (local address 127.0.0.1:47512)
0925 05:00:23.779081 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:23.779180 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:40077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260925 05:00:23.780542 29217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:23.780163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:40077 (local address 127.0.0.1:47522)
0925 05:00:23.780359 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:23.780429 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:40077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260925 05:00:23.780771 29215 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.29.188:40077: Network error: Client connection negotiation failed: client connection to 127.6.29.188:40077: connect: Connection refused (error 111)
I20260925 05:00:23.789695 29141 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.29.189" port: 41607 } attempt: 5
Sep 25 05:00:23 dist-test-slave-xzjr krb5kdc[29061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312423, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for kudu/127.6.29.189@KRBTEST.COM
I20260925 05:00:23.803710 29219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:23.790515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34016 (local address 127.6.29.189:41607)
0925 05:00:23.790701 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:23.790706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:23.790730 (+ 24us) server_negotiation.cc:407] Connection header received
0925 05:00:23.790911 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:23.790916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:23.791016 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:23.791146 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:23.791966 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:23.793020 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:23.794193 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:23.794657 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:23.794808 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:23.798339 (+ 3531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:23.798344 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:23.798347 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:23.798380 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:23.801104 (+ 2724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:23.801768 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:23.801775 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:23.801778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:23.801900 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:23.802282 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:23.802288 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:23.802292 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:23.802586 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:23.802746 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:23.803134 (+ 388us) server_negotiation.cc:299] Negotiation successful
0925 05:00:23.803368 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 05:00:23.803728 29144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:23.790437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:41607 (local address 127.0.0.1:34016)
0925 05:00:23.790647 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:23.790666 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:23.790782 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:23.791193 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:23.791196 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:23.791580 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:00:23.791803 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:23.791824 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:23.793180 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:23.793183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:23.794038 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:23.794050 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:23.794833 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:23.794840 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:23.795033 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:23.795730 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:00:23.795748 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:00:23.798129 (+ 2381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:00:23.801260 (+ 3131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:23.801265 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:23.801270 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:23.801639 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:23.802016 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:23.802021 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:23.802025 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:23.802172 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:23.802716 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:00:23.802721 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:00:23.802882 (+ 161us) client_negotiation.cc:769] Sending connection context
0925 05:00:23.803034 (+ 152us) client_negotiation.cc:240] Negotiation successful
0925 05:00:23.803340 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 05:00:23.804973 29141 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.29.188" port: 40077 } has no permanent_uuid. Determining permanent_uuid...
I20260925 05:00:23.806051 29144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:23.805654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:40077 (local address 127.0.0.1:47524)
0925 05:00:23.805829 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:23.805902 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:40077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 05:00:23.807205 29144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:23.806754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:40077 (local address 127.0.0.1:47528)
0925 05:00:23.807018 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:23.807091 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:40077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
W20260925 05:00:23.807425 29141 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.29.188:40077: Network error: Client connection negotiation failed: client connection to 127.6.29.188:40077: connect: Connection refused (error 111)
I20260925 05:00:23.858668 29141 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.29.188" port: 40077 } attempt: 1
I20260925 05:00:23.859968 29144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:23.859400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:40077 (local address 127.0.0.1:47534)
0925 05:00:23.859608 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:23.859715 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:40077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 05:00:23.861028 29144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:23.860690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:40077 (local address 127.0.0.1:47536)
0925 05:00:23.860837 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:23.860916 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:40077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 05:00:23.861256 29141 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.29.188:40077: Network error: Client connection negotiation failed: client connection to 127.6.29.188:40077: connect: Connection refused (error 111)
I20260925 05:00:23.871990 29215 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.29.188" port: 40077 } attempt: 3
I20260925 05:00:23.873191 29217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:23.872715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:40077 (local address 127.0.0.1:47552)
0925 05:00:23.872924 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:23.873022 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:40077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 05:00:23.874322 29217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:23.873919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:40077 (local address 127.0.0.1:47560)
0925 05:00:23.874134 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:23.874215 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:40077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
W20260925 05:00:23.874564 29215 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.29.188:40077: Network error: Client connection negotiation failed: client connection to 127.6.29.188:40077: connect: Connection refused (error 111)
I20260925 05:00:23.929492 29141 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.29.188" port: 40077 } attempt: 2
I20260925 05:00:23.930869 29144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:23.930345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:40077 (local address 127.0.0.1:47566)
0925 05:00:23.930607 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:23.930697 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:40077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260925 05:00:23.932009 29144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:23.931637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:40077 (local address 127.0.0.1:47574)
0925 05:00:23.931810 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:23.931899 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:40077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 05:00:23.932222 29141 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.29.188:40077: Network error: Client connection negotiation failed: client connection to 127.6.29.188:40077: connect: Connection refused (error 111)
W20260925 05:00:24.010409 29222 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 05:00:24.010792 29222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 05:00:24.010918 29222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 05:00:24.011062 29222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 05:00:24.017789 29215 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.29.188" port: 40077 } attempt: 4
W20260925 05:00:24.018877 29222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 05:00:24.018996 29222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 05:00:24.019124 29222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
I20260925 05:00:24.019099 29217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:24.018545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:40077 (local address 127.0.0.1:47582)
0925 05:00:24.018798 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:24.018891 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:40077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260925 05:00:24.019215 29222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 05:00:24.019306 29222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 05:00:24.020206 29217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:24.019889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:40077 (local address 127.0.0.1:47594)
0925 05:00:24.020029 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:24.020102 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:40077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 05:00:24.020412 29215 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.29.188:40077: Network error: Client connection negotiation failed: client connection to 127.6.29.188:40077: connect: Connection refused (error 111)
I20260925 05:00:24.029798 29222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.29.148:40773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.29.190:43911,127.6.29.189:41607,127.6.29.188:40077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/krb5kdc/kudu_127.6.29.188.keytab
--principal=kudu/127.6.29.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.29.188:40077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.29.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 04:03:47 UTC on ccycloud.quasar-nljujk.root.comops.site
build id 4671827
ASAN enabled
I20260925 05:00:24.031210 29222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 05:00:24.032697 29222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 05:00:24.038470 29141 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.29.188" port: 40077 } attempt: 3
I20260925 05:00:24.039911 29144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:24.039238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:40077 (local address 127.0.0.1:47610)
0925 05:00:24.039497 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:24.039637 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:40077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 05:00:24.040922 29144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:24.040529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:40077 (local address 127.0.0.1:47620)
0925 05:00:24.040751 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:24.040820 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.29.188:40077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260925 05:00:24.041121 29141 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.29.188:40077: Network error: Client connection negotiation failed: client connection to 127.6.29.188:40077: connect: Connection refused (error 111)
W20260925 05:00:24.045702 29227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:24.045646 29228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 05:00:24.045725 29230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 05:00:24.046764 29222 server_base.cc:1034] running on GCE node
I20260925 05:00:24.047598 29222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 05:00:24.049067 29222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 05:00:24.050318 29222 hybrid_clock.cc:648] HybridClock initialized: now 1790312424050232 us; error 60 us; skew 500 ppm
I20260925 05:00:24.050663 29222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 05:00:24 dist-test-slave-xzjr krb5kdc[29061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312424, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 05:00:24.053761 29222 init.cc:377] Logged in from keytab as kudu/127.6.29.188@KRBTEST.COM (short username kudu)
I20260925 05:00:24.055255 29222 webserver.cc:460] Webserver started at http://127.6.29.188:39643/ using document root <none> and password file <none>
I20260925 05:00:24.056053 29222 fs_manager.cc:359] Metadata directory not provided
I20260925 05:00:24.056185 29222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 05:00:24.056463 29222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 05:00:24.059437 29222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-2/data/instance:
uuid: "2ad3b8e74ffc47658ba794e55ab8ba20"
format_stamp: "Formatted at 2026-09-25 05:00:24 on dist-test-slave-xzjr"
server_key: "d711f4d53ab5ae8e32439a0430b0a949"
server_key_iv: "60fa5b4d20e151c261126c01dd17bc0f"
server_key_version: "encryptionkey@0"
I20260925 05:00:24.060253 29222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-2/wal/instance:
uuid: "2ad3b8e74ffc47658ba794e55ab8ba20"
format_stamp: "Formatted at 2026-09-25 05:00:24 on dist-test-slave-xzjr"
server_key: "d711f4d53ab5ae8e32439a0430b0a949"
server_key_iv: "60fa5b4d20e151c261126c01dd17bc0f"
server_key_version: "encryptionkey@0"
I20260925 05:00:24.065480 29222 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 05:00:24.068907 29237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 05:00:24.070302 29222 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 05:00:24.070442 29222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-2/data,/tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-2/wal
uuid: "2ad3b8e74ffc47658ba794e55ab8ba20"
format_stamp: "Formatted at 2026-09-25 05:00:24 on dist-test-slave-xzjr"
server_key: "d711f4d53ab5ae8e32439a0430b0a949"
server_key_iv: "60fa5b4d20e151c261126c01dd17bc0f"
server_key_version: "encryptionkey@0"
I20260925 05:00:24.070631 29222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 05:00:24.086833 29222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 05:00:24.091696 29222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 05:00:24.091998 29222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 05:00:24.109786 29222 rpc_server.cc:307] RPC server started. Bound to: 127.6.29.188:40077
I20260925 05:00:24.109807 29288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.29.188:40077 every 8 connection(s)
I20260925 05:00:24.111217 29222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-2/data/info.pb
I20260925 05:00:24.115217 29289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 05:00:24.116408 6262 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu as pid 29222
I20260925 05:00:24.116595 6262 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskjC0Yza/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790312053666503-6262-0/minicluster-data/master-2/wal/instance
I20260925 05:00:24.117139 6262 external_mini_cluster.cc:1428] Setting key fd3bdeff109f84a41869b02e1a9a8363
I20260925 05:00:24.125844 29289 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43911 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:00:24 dist-test-slave-xzjr krb5kdc[29061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.188@KRBTEST.COM
Sep 25 05:00:24 dist-test-slave-xzjr krb5kdc[29061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312424, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.188@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
I20260925 05:00:24.142416 29292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:24.119100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47634 (local address 127.6.29.188:40077)
0925 05:00:24.119930 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:24.119946 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:24.119995 (+ 49us) server_negotiation.cc:407] Connection header received
0925 05:00:24.122520 (+ 2525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:24.122552 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:24.123140 (+ 588us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:24.123610 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:24.125007 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:24.127109 (+ 2102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:24.128571 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:24.129493 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:24.129697 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:24.134283 (+ 4586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:24.134294 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:24.134311 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:24.134365 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:24.137999 (+ 3634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:24.138883 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:24.138889 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:24.138896 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:24.139038 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:24.139441 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:24.139445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:24.139448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:24.140014 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:24.140328 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:24.140764 (+ 436us) server_negotiation.cc:299] Negotiation successful
0925 05:00:24.141192 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":347,"threads_started":1}
I20260925 05:00:24.144812 29293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:24.128968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.190:43911 (local address 127.0.0.1:48434)
0925 05:00:24.129907 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:24.129942 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:24.130113 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:24.130546 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:24.130551 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:24.131282 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:00:24.131524 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:24.131532 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:24.132977 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:24.132984 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:24.134169 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:24.134181 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:24.134931 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:24.134936 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:24.135137 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:24.135870 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:00:24.135902 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:00:24.138693 (+ 2791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:00:24.141889 (+ 3196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:24.141898 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:24.141905 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:24.142428 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:24.142879 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:24.142887 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:24.142893 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:24.143062 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:24.143598 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:00:24.143606 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:00:24.143857 (+ 251us) client_negotiation.cc:769] Sending connection context
0925 05:00:24.144039 (+ 182us) client_negotiation.cc:240] Negotiation successful
0925 05:00:24.144349 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"thread_start_us":444,"threads_started":1}
I20260925 05:00:24.144716 29143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:24.128963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48434 (local address 127.6.29.190:43911)
0925 05:00:24.129203 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:24.129211 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:24.130043 (+ 832us) server_negotiation.cc:407] Connection header received
0925 05:00:24.130269 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:24.130274 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:24.130367 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:24.130478 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:24.131673 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:24.132790 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:24.134322 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:24.134744 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:24.134837 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:24.138897 (+ 4060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:24.138902 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:24.138906 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:24.138940 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:24.141765 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:24.142578 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:24.142585 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:24.142590 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:24.142749 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:24.143185 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:24.143191 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:24.143194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:24.143461 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:24.143625 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:24.144158 (+ 533us) server_negotiation.cc:299] Negotiation successful
0925 05:00:24.144362 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 05:00:24.146396 29289 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.29.189" port: 41607 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:00:24 dist-test-slave-xzjr krb5kdc[29061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.190@KRBTEST.COM
Sep 25 05:00:24 dist-test-slave-xzjr krb5kdc[29061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312424, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.188@KRBTEST.COM for kudu/127.6.29.189@KRBTEST.COM
I20260925 05:00:24.161202 29293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:24.147170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.189:41607 (local address 127.0.0.1:34022)
0925 05:00:24.147439 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:24.147453 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:24.147575 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:24.147960 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:24.147965 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:24.148320 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:00:24.148450 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:24.148458 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:24.149403 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:24.149410 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:24.150273 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:24.150284 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:24.151130 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:24.151136 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:24.151328 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:24.151917 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:00:24.151933 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:00:24.154403 (+ 2470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:00:24.158305 (+ 3902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:24.158310 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:24.158314 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:24.158821 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:24.159335 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:24.159340 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:24.159344 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:24.159555 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:24.160226 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:00:24.160230 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:00:24.160404 (+ 174us) client_negotiation.cc:769] Sending connection context
0925 05:00:24.160556 (+ 152us) client_negotiation.cc:240] Negotiation successful
0925 05:00:24.160860 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188}
I20260925 05:00:24.161276 29143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:24.145685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48448 (local address 127.6.29.190:43911)
0925 05:00:24.145827 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:24.145831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:24.145843 (+ 12us) server_negotiation.cc:407] Connection header received
0925 05:00:24.145879 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:24.145883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:24.145931 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:24.146001 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:24.147456 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:24.148288 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:24.149521 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:24.149935 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:24.150070 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:24.154330 (+ 4260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:24.154335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:24.154338 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:24.154369 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:24.158098 (+ 3729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:24.159026 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:24.159032 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:24.159035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:24.159214 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:24.159669 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:24.159674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:24.159678 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:24.160058 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:24.160367 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:24.160674 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 05:00:24.160907 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 05:00:24.161850 29219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:24.147174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34022 (local address 127.6.29.189:41607)
0925 05:00:24.147338 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:24.147342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:24.147544 (+ 202us) server_negotiation.cc:407] Connection header received
0925 05:00:24.147705 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:24.147711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:24.147830 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:24.147941 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:24.148601 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:24.149269 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:24.150411 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:24.150973 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:24.151118 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:24.154621 (+ 3503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:24.154628 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:24.154635 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:24.154673 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:24.158169 (+ 3496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:24.159036 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:24.159041 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:24.159044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:24.159200 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:24.159669 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:24.159674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:24.159678 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:24.160059 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:24.160238 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:24.160680 (+ 442us) server_negotiation.cc:299] Negotiation successful
0925 05:00:24.160903 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 05:00:24.162853 29289 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.29.188" port: 40077 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 05:00:24 dist-test-slave-xzjr krb5kdc[29061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312424, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.188@KRBTEST.COM for kudu/127.6.29.188@KRBTEST.COM
Sep 25 05:00:24 dist-test-slave-xzjr krb5kdc[29061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.29.189@KRBTEST.COM
I20260925 05:00:24.178339 29293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:24.163535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:40077 (local address 127.0.0.1:47648)
0925 05:00:24.163833 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:24.163851 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:24.163957 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:24.164264 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:24.164270 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:24.164768 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:00:24.164953 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:24.164964 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:24.166275 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:24.166278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:24.167656 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:24.167673 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:24.168613 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:24.168617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:24.168881 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:24.169564 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:00:24.169588 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:00:24.172231 (+ 2643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:00:24.175857 (+ 3626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:24.175861 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:24.175864 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:24.176212 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:24.176582 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:24.176585 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:24.176587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:24.176711 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:24.177307 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:00:24.177312 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:00:24.177480 (+ 168us) client_negotiation.cc:769] Sending connection context
0925 05:00:24.177643 (+ 163us) client_negotiation.cc:240] Negotiation successful
0925 05:00:24.177915 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165}
I20260925 05:00:24.178432 29292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:24.163575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47648 (local address 127.6.29.188:40077)
0925 05:00:24.163742 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:24.163747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:24.163919 (+ 172us) server_negotiation.cc:407] Connection header received
0925 05:00:24.164056 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:24.164059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:24.164111 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:24.164203 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:24.165087 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:24.166143 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:24.167845 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:24.168452 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:24.168577 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:24.172399 (+ 3822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:24.172405 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:24.172411 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:24.172449 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:24.175693 (+ 3244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:24.176330 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:24.176338 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:24.176341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:24.176473 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:24.176814 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:24.176821 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:24.176827 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:24.177195 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:24.177297 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:24.177747 (+ 450us) server_negotiation.cc:299] Negotiation successful
0925 05:00:24.178142 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":109,"server-negotiator.queue_time_us":60}
I20260925 05:00:24.179891 29219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:24.162985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34038 (local address 127.6.29.189:41607)
0925 05:00:24.163217 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:24.163224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:24.163243 (+ 19us) server_negotiation.cc:407] Connection header received
0925 05:00:24.163424 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:24.163430 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:24.163502 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:24.163595 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:24.165266 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:24.166299 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:24.167967 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:24.168584 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:24.168717 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:24.173496 (+ 4779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:24.173501 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:24.173504 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:24.173533 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:24.176968 (+ 3435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:24.177751 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:24.177757 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:24.177760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:24.177936 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:24.178380 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:24.178386 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:24.178389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:24.178730 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:24.179014 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:24.179283 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 05:00:24.179506 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260925 05:00:24.184342 29289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ad3b8e74ffc47658ba794e55ab8ba20: Bootstrap starting.
I20260925 05:00:24.184592 6262 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 05:00:24.187983 29289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ad3b8e74ffc47658ba794e55ab8ba20: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:24.189339 29289 log.cc:824] T 00000000000000000000000000000000 P 2ad3b8e74ffc47658ba794e55ab8ba20: Log is configured to *not* fsync() on all Append() calls
I20260925 05:00:24.187022 29116 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "75b663e2c09043658a3ace1ca5558b51"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48448
I20260925 05:00:24.192562 29289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ad3b8e74ffc47658ba794e55ab8ba20: No bootstrap required, opened a new log
I20260925 05:00:24.203430 29289 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ad3b8e74ffc47658ba794e55ab8ba20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b663e2c09043658a3ace1ca5558b51" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43911 } } peers { permanent_uuid: "fba2a365442d40c7b44cf58d6a639540" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 41607 } } peers { permanent_uuid: "2ad3b8e74ffc47658ba794e55ab8ba20" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 40077 } }
I20260925 05:00:24.203910 29289 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ad3b8e74ffc47658ba794e55ab8ba20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ad3b8e74ffc47658ba794e55ab8ba20, State: Initialized, Role: FOLLOWER
I20260925 05:00:24.204440 29289 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ad3b8e74ffc47658ba794e55ab8ba20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b663e2c09043658a3ace1ca5558b51" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43911 } } peers { permanent_uuid: "fba2a365442d40c7b44cf58d6a639540" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 41607 } } peers { permanent_uuid: "2ad3b8e74ffc47658ba794e55ab8ba20" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 40077 } }
I20260925 05:00:24.205399 29295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ad3b8e74ffc47658ba794e55ab8ba20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b663e2c09043658a3ace1ca5558b51" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43911 } } peers { permanent_uuid: "fba2a365442d40c7b44cf58d6a639540" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 41607 } } peers { permanent_uuid: "2ad3b8e74ffc47658ba794e55ab8ba20" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 40077 } } }
I20260925 05:00:24.205744 29295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ad3b8e74ffc47658ba794e55ab8ba20 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:00:24.206722 29289 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ad3b8e74ffc47658ba794e55ab8ba20 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 05:00:24.216280 29306 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2ad3b8e74ffc47658ba794e55ab8ba20: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:00:24.216434 29306 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:00:24.218354 29141 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.29.188" port: 40077 } attempt: 4
I20260925 05:00:24.218246 29222 master_runner.cc:186] Error getting master registration for 127.6.29.190:43911: OK, instance_id { permanent_uuid: "75b663e2c09043658a3ace1ca5558b51" instance_seqno: 1790312423077059 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 05:00:24.229630 29222 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 25 05:00:24 dist-test-slave-xzjr krb5kdc[29061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312423, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.190@KRBTEST.COM for kudu/127.6.29.188@KRBTEST.COM
I20260925 05:00:24.238520 29292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:24.221451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47650 (local address 127.6.29.188:40077)
0925 05:00:24.221669 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:24.221680 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:24.221703 (+ 23us) server_negotiation.cc:407] Connection header received
0925 05:00:24.221762 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:24.221767 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:24.221852 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:24.221959 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:24.223013 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:24.224318 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:24.226227 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:24.226961 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:24.227090 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:24.231536 (+ 4446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:24.231543 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:24.231549 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:24.231596 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:24.235340 (+ 3744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:24.236269 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:24.236275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:24.236279 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:24.236427 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:24.236869 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:24.236875 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:24.236879 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:24.237289 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:24.237461 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:24.237882 (+ 421us) server_negotiation.cc:299] Negotiation successful
0925 05:00:24.238132 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 05:00:24.238708 29144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:24.218995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:40077 (local address 127.0.0.1:47650)
0925 05:00:24.219383 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:24.219403 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:24.219519 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:24.222040 (+ 2521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:24.222046 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:24.222575 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:00:24.222814 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:24.222828 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:24.224530 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:24.224536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:24.226039 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:24.226055 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:24.227378 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:24.227385 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:24.227593 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:24.228345 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:00:24.228383 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:00:24.231349 (+ 2966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:00:24.235539 (+ 4190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:24.235546 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:24.235550 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:24.236122 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:24.236570 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:24.236574 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:24.236579 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:24.236752 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:24.237418 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:00:24.237423 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:00:24.237607 (+ 184us) client_negotiation.cc:769] Sending connection context
0925 05:00:24.237751 (+ 144us) client_negotiation.cc:240] Negotiation successful
0925 05:00:24.238066 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244}
I20260925 05:00:24.243436 29141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75b663e2c09043658a3ace1ca5558b51: Bootstrap starting.
I20260925 05:00:24.247509 29141 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75b663e2c09043658a3ace1ca5558b51: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:24.248852 29141 log.cc:824] T 00000000000000000000000000000000 P 75b663e2c09043658a3ace1ca5558b51: Log is configured to *not* fsync() on all Append() calls
I20260925 05:00:24.252526 29141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75b663e2c09043658a3ace1ca5558b51: No bootstrap required, opened a new log
I20260925 05:00:24.260347 29141 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75b663e2c09043658a3ace1ca5558b51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b663e2c09043658a3ace1ca5558b51" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43911 } } peers { permanent_uuid: "fba2a365442d40c7b44cf58d6a639540" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 41607 } } peers { permanent_uuid: "2ad3b8e74ffc47658ba794e55ab8ba20" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 40077 } }
I20260925 05:00:24.260794 29141 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75b663e2c09043658a3ace1ca5558b51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75b663e2c09043658a3ace1ca5558b51, State: Initialized, Role: FOLLOWER
I20260925 05:00:24.261315 29141 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75b663e2c09043658a3ace1ca5558b51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b663e2c09043658a3ace1ca5558b51" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43911 } } peers { permanent_uuid: "fba2a365442d40c7b44cf58d6a639540" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 41607 } } peers { permanent_uuid: "2ad3b8e74ffc47658ba794e55ab8ba20" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 40077 } }
I20260925 05:00:24.262382 29309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75b663e2c09043658a3ace1ca5558b51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b663e2c09043658a3ace1ca5558b51" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43911 } } peers { permanent_uuid: "fba2a365442d40c7b44cf58d6a639540" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 41607 } } peers { permanent_uuid: "2ad3b8e74ffc47658ba794e55ab8ba20" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 40077 } } }
I20260925 05:00:24.262770 29309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75b663e2c09043658a3ace1ca5558b51 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:00:24.263692 29141 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75b663e2c09043658a3ace1ca5558b51 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:00:24.263824 29116 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75b663e2c09043658a3ace1ca5558b51 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 05:00:24.264048 29116 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75b663e2c09043658a3ace1ca5558b51 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:24.266944 29116 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75b663e2c09043658a3ace1ca5558b51 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b663e2c09043658a3ace1ca5558b51" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43911 } } peers { permanent_uuid: "fba2a365442d40c7b44cf58d6a639540" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 41607 } } peers { permanent_uuid: "2ad3b8e74ffc47658ba794e55ab8ba20" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 40077 } }
I20260925 05:00:24.268325 29116 leader_election.cc:288] T 00000000000000000000000000000000 P 75b663e2c09043658a3ace1ca5558b51 [CANDIDATE]: Term 1 election: Requested vote from peers fba2a365442d40c7b44cf58d6a639540 (127.6.29.189:41607), 2ad3b8e74ffc47658ba794e55ab8ba20 (127.6.29.188:40077)
I20260925 05:00:24.269279 29264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "75b663e2c09043658a3ace1ca5558b51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2ad3b8e74ffc47658ba794e55ab8ba20"
I20260925 05:00:24.269771 29264 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ad3b8e74ffc47658ba794e55ab8ba20 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:24.273252 29264 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2ad3b8e74ffc47658ba794e55ab8ba20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75b663e2c09043658a3ace1ca5558b51 in term 1.
I20260925 05:00:24.273869 29092 leader_election.cc:302] T 00000000000000000000000000000000 P 75b663e2c09043658a3ace1ca5558b51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ad3b8e74ffc47658ba794e55ab8ba20, 75b663e2c09043658a3ace1ca5558b51; no voters:
I20260925 05:00:24.274361 29309 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75b663e2c09043658a3ace1ca5558b51 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 05:00:24.285543 29309 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75b663e2c09043658a3ace1ca5558b51 [term 1 LEADER]: Becoming Leader. State: Replica: 75b663e2c09043658a3ace1ca5558b51, State: Running, Role: LEADER
I20260925 05:00:24.287060 29309 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75b663e2c09043658a3ace1ca5558b51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b663e2c09043658a3ace1ca5558b51" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43911 } } peers { permanent_uuid: "fba2a365442d40c7b44cf58d6a639540" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 41607 } } peers { permanent_uuid: "2ad3b8e74ffc47658ba794e55ab8ba20" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 40077 } }
W20260925 05:00:24.316715 29095 tablet.cc:2369] T 00000000000000000000000000000000 P 75b663e2c09043658a3ace1ca5558b51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 05:00:24.319582 29215 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.29.188" port: 40077 } attempt: 5
I20260925 05:00:24.330662 29317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75b663e2c09043658a3ace1ca5558b51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75b663e2c09043658a3ace1ca5558b51. Latest consensus state: current_term: 1 leader_uuid: "75b663e2c09043658a3ace1ca5558b51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b663e2c09043658a3ace1ca5558b51" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43911 } } peers { permanent_uuid: "fba2a365442d40c7b44cf58d6a639540" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 41607 } } peers { permanent_uuid: "2ad3b8e74ffc47658ba794e55ab8ba20" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 40077 } } }
I20260925 05:00:24.331292 29317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75b663e2c09043658a3ace1ca5558b51 [sys.catalog]: This master's current role is: LEADER
I20260925 05:00:24.332129 29143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:24.299395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48456 (local address 127.6.29.190:43911)
0925 05:00:24.299622 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:24.299642 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:24.308879 (+ 9237us) server_negotiation.cc:407] Connection header received
0925 05:00:24.309210 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:24.309220 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:24.309318 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:24.309412 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:24.313989 (+ 4577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:24.315166 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:24.317232 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:24.317959 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:24.318064 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:24.323849 (+ 5785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:24.323859 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:24.323865 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:24.323910 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:24.327569 (+ 3659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:24.328658 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:24.328665 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:24.328668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:24.328951 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:24.329622 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:24.329628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:24.329632 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:24.330007 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:24.330290 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:24.330894 (+ 604us) server_negotiation.cc:299] Negotiation successful
0925 05:00:24.331159 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260925 05:00:24.344029 29243 tablet.cc:2369] T 00000000000000000000000000000000 P 2ad3b8e74ffc47658ba794e55ab8ba20: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 05:00:24 dist-test-slave-xzjr krb5kdc[29061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790312423, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.29.189@KRBTEST.COM for kudu/127.6.29.188@KRBTEST.COM
I20260925 05:00:24.353673 29217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:24.320284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.29.188:40077 (local address 127.0.0.1:47676)
0925 05:00:24.320617 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0925 05:00:24.320637 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 05:00:24.320774 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 05:00:24.337204 (+ 16430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 05:00:24.337212 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 05:00:24.337842 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0925 05:00:24.338099 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:24.338113 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:24.340041 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:24.340046 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:24.341278 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 05:00:24.341294 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 05:00:24.342443 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 05:00:24.342450 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 05:00:24.342698 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:24.343412 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 05:00:24.343439 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 05:00:24.346357 (+ 2918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 05:00:24.350802 (+ 4445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:24.350808 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:24.350812 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:24.351329 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:24.351846 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 05:00:24.351851 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 05:00:24.351855 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 05:00:24.352028 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 05:00:24.352671 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 05:00:24.352677 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 05:00:24.352842 (+ 165us) client_negotiation.cc:769] Sending connection context
0925 05:00:24.352972 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 05:00:24.353262 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194}
I20260925 05:00:24.354811 29329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:24.320354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47676 (local address 127.6.29.188:40077)
0925 05:00:24.336583 (+ 16229us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:24.336592 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:24.336623 (+ 31us) server_negotiation.cc:407] Connection header received
0925 05:00:24.336713 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:24.336722 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:24.336836 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:24.338278 (+ 1442us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:24.338642 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:24.339908 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:24.341590 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:24.342271 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:24.343060 (+ 789us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:24.346615 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:24.346622 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:24.346636 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:24.346679 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:24.350650 (+ 3971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:24.351495 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:24.351502 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:24.351507 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:24.351692 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:24.352167 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:24.352172 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:24.352175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:24.352523 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:24.353937 (+ 1414us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:24.354065 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 05:00:24.354338 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16005,"thread_start_us":16615,"threads_started":1}
W20260925 05:00:24.360442 29330 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 75b663e2c09043658a3ace1ca5558b51: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 05:00:24.360756 29330 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:00:24.360913 29215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fba2a365442d40c7b44cf58d6a639540: Bootstrap starting.
I20260925 05:00:24.363204 29331 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 05:00:24.364675 29292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:24.307676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47666 (local address 127.6.29.188:40077)
0925 05:00:24.307916 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:24.307924 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:24.320278 (+ 12354us) server_negotiation.cc:407] Connection header received
0925 05:00:24.322468 (+ 2190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:24.322476 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:24.322559 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:24.322634 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:24.325895 (+ 3261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:24.327368 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:24.340604 (+ 13236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:24.341296 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:24.341426 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:24.352565 (+ 11139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:24.352572 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:24.352577 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:24.352617 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:24.356482 (+ 3865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:24.359670 (+ 3188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:24.359676 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:24.359681 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:24.359874 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:24.362363 (+ 2489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:24.362369 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:24.362373 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:24.362759 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:24.363010 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:24.364051 (+ 1041us) server_negotiation.cc:299] Negotiation successful
0925 05:00:24.364297 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 05:00:24.365394 29215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fba2a365442d40c7b44cf58d6a639540: Neither blocks nor log segments found. Creating new log.
I20260925 05:00:24.367364 29215 log.cc:824] T 00000000000000000000000000000000 P fba2a365442d40c7b44cf58d6a639540: Log is configured to *not* fsync() on all Append() calls
I20260925 05:00:24.347244 29074 master_runner.cc:186] Error getting master registration for 127.6.29.189:41607: OK, instance_id { permanent_uuid: "fba2a365442d40c7b44cf58d6a639540" instance_seqno: 1790312423587712 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 05:00:24.369892 29074 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 05:00:24.374742 29331 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 05:00:24.377310 29215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fba2a365442d40c7b44cf58d6a639540: No bootstrap required, opened a new log
I20260925 05:00:24.379575 29219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:24.301932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34052 (local address 127.6.29.189:41607)
0925 05:00:24.302194 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:24.302202 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:24.319393 (+ 17191us) server_negotiation.cc:407] Connection header received
0925 05:00:24.321731 (+ 2338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:24.321739 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:24.321841 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:24.321926 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:24.342183 (+ 20257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:24.343276 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:24.354326 (+ 11050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:24.355036 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:24.355189 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:24.368586 (+ 13397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 05:00:24.368596 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 05:00:24.368603 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 05:00:24.368662 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0925 05:00:24.374186 (+ 5524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:24.375701 (+ 1515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:24.375722 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:24.375726 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:24.375998 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 05:00:24.376722 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 05:00:24.376726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 05:00:24.376750 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0925 05:00:24.377301 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 05:00:24.377633 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:24.378610 (+ 977us) server_negotiation.cc:299] Negotiation successful
0925 05:00:24.378986 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 05:00:24.388878 29215 raft_consensus.cc:348] T 00000000000000000000000000000000 P fba2a365442d40c7b44cf58d6a639540 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b663e2c09043658a3ace1ca5558b51" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43911 } } peers { permanent_uuid: "fba2a365442d40c7b44cf58d6a639540" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 41607 } } peers { permanent_uuid: "2ad3b8e74ffc47658ba794e55ab8ba20" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 40077 } }
I20260925 05:00:24.389361 29215 raft_consensus.cc:729] T 00000000000000000000000000000000 P fba2a365442d40c7b44cf58d6a639540 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fba2a365442d40c7b44cf58d6a639540, State: Initialized, Role: FOLLOWER
I20260925 05:00:24.389963 29215 consensus_queue.cc:260] T 00000000000000000000000000000000 P fba2a365442d40c7b44cf58d6a639540 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b663e2c09043658a3ace1ca5558b51" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43911 } } peers { permanent_uuid: "fba2a365442d40c7b44cf58d6a639540" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 41607 } } peers { permanent_uuid: "2ad3b8e74ffc47658ba794e55ab8ba20" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 40077 } }
I20260925 05:00:24.391369 29335 sys_catalog.cc:455] T 00000000000000000000000000000000 P fba2a365442d40c7b44cf58d6a639540 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b663e2c09043658a3ace1ca5558b51" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43911 } } peers { permanent_uuid: "fba2a365442d40c7b44cf58d6a639540" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 41607 } } peers { permanent_uuid: "2ad3b8e74ffc47658ba794e55ab8ba20" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 40077 } } }
I20260925 05:00:24.392025 29335 sys_catalog.cc:458] T 00000000000000000000000000000000 P fba2a365442d40c7b44cf58d6a639540 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:00:24.392695 29215 sys_catalog.cc:564] T 00000000000000000000000000000000 P fba2a365442d40c7b44cf58d6a639540 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 05:00:24.392719 29190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "75b663e2c09043658a3ace1ca5558b51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fba2a365442d40c7b44cf58d6a639540"
I20260925 05:00:24.393186 29190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fba2a365442d40c7b44cf58d6a639540 [term 0 FOLLOWER]: Advancing to term 1
I20260925 05:00:24.406016 29264 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2ad3b8e74ffc47658ba794e55ab8ba20 [term 1 FOLLOWER]: Refusing update from remote peer 75b663e2c09043658a3ace1ca5558b51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:00:24.409515 29317 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 75b663e2c09043658a3ace1ca5558b51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ad3b8e74ffc47658ba794e55ab8ba20" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 40077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:00:24.417671 29190 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fba2a365442d40c7b44cf58d6a639540 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75b663e2c09043658a3ace1ca5558b51 in term 1.
I20260925 05:00:24.418725 29189 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fba2a365442d40c7b44cf58d6a639540 [term 1 FOLLOWER]: Refusing update from remote peer 75b663e2c09043658a3ace1ca5558b51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 05:00:24.423796 29317 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 75b663e2c09043658a3ace1ca5558b51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fba2a365442d40c7b44cf58d6a639540" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 41607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 05:00:24.445528 29335 sys_catalog.cc:455] T 00000000000000000000000000000000 P fba2a365442d40c7b44cf58d6a639540 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75b663e2c09043658a3ace1ca5558b51. Latest consensus state: current_term: 1 leader_uuid: "75b663e2c09043658a3ace1ca5558b51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b663e2c09043658a3ace1ca5558b51" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43911 } } peers { permanent_uuid: "fba2a365442d40c7b44cf58d6a639540" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 41607 } } peers { permanent_uuid: "2ad3b8e74ffc47658ba794e55ab8ba20" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 40077 } } }
I20260925 05:00:24.445946 29335 sys_catalog.cc:458] T 00000000000000000000000000000000 P fba2a365442d40c7b44cf58d6a639540 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:00:24.447721 29295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ad3b8e74ffc47658ba794e55ab8ba20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75b663e2c09043658a3ace1ca5558b51. Latest consensus state: current_term: 1 leader_uuid: "75b663e2c09043658a3ace1ca5558b51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b663e2c09043658a3ace1ca5558b51" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43911 } } peers { permanent_uuid: "fba2a365442d40c7b44cf58d6a639540" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 41607 } } peers { permanent_uuid: "2ad3b8e74ffc47658ba794e55ab8ba20" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 40077 } } }
I20260925 05:00:24.448079 29295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ad3b8e74ffc47658ba794e55ab8ba20 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:00:24.451695 29334 mvcc.cc:205] Tried to move back new op lower bound from 7333119690321563648 to 7333119689897316352. Current Snapshot: MvccSnapshot[applied={T|T < 7333119690321563648}]
I20260925 05:00:24.461367 29344 mvcc.cc:205] Tried to move back new op lower bound from 7333119690321563648 to 7333119689897316352. Current Snapshot: MvccSnapshot[applied={T|T < 7333119690321563648}]
I20260925 05:00:24.468160 29309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75b663e2c09043658a3ace1ca5558b51 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "75b663e2c09043658a3ace1ca5558b51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b663e2c09043658a3ace1ca5558b51" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43911 } } peers { permanent_uuid: "fba2a365442d40c7b44cf58d6a639540" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 41607 } } peers { permanent_uuid: "2ad3b8e74ffc47658ba794e55ab8ba20" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 40077 } } }
I20260925 05:00:24.468501 29309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75b663e2c09043658a3ace1ca5558b51 [sys.catalog]: This master's current role is: LEADER
I20260925 05:00:24.468681 29309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75b663e2c09043658a3ace1ca5558b51 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "75b663e2c09043658a3ace1ca5558b51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b663e2c09043658a3ace1ca5558b51" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43911 } } peers { permanent_uuid: "fba2a365442d40c7b44cf58d6a639540" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 41607 } } peers { permanent_uuid: "2ad3b8e74ffc47658ba794e55ab8ba20" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 40077 } } }
I20260925 05:00:24.468890 29309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75b663e2c09043658a3ace1ca5558b51 [sys.catalog]: This master's current role is: LEADER
I20260925 05:00:24.475178 29331 catalog_manager.cc:1347] Generated new cluster ID: 67449104075a4d6d9582cf5d602e8cab
I20260925 05:00:24.475281 29331 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 05:00:24.474561 29335 sys_catalog.cc:455] T 00000000000000000000000000000000 P fba2a365442d40c7b44cf58d6a639540 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "75b663e2c09043658a3ace1ca5558b51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b663e2c09043658a3ace1ca5558b51" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43911 } } peers { permanent_uuid: "fba2a365442d40c7b44cf58d6a639540" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 41607 } } peers { permanent_uuid: "2ad3b8e74ffc47658ba794e55ab8ba20" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 40077 } } }
I20260925 05:00:24.475843 29335 sys_catalog.cc:458] T 00000000000000000000000000000000 P fba2a365442d40c7b44cf58d6a639540 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:00:24.488375 29295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ad3b8e74ffc47658ba794e55ab8ba20 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "75b663e2c09043658a3ace1ca5558b51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b663e2c09043658a3ace1ca5558b51" member_type: VOTER last_known_addr { host: "127.6.29.190" port: 43911 } } peers { permanent_uuid: "fba2a365442d40c7b44cf58d6a639540" member_type: VOTER last_known_addr { host: "127.6.29.189" port: 41607 } } peers { permanent_uuid: "2ad3b8e74ffc47658ba794e55ab8ba20" member_type: VOTER last_known_addr { host: "127.6.29.188" port: 40077 } } }
I20260925 05:00:24.488787 29295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ad3b8e74ffc47658ba794e55ab8ba20 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 05:00:24.523749 29352 catalog_manager.cc:1259] Loaded cluster ID: 67449104075a4d6d9582cf5d602e8cab
I20260925 05:00:24.523975 29352 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fba2a365442d40c7b44cf58d6a639540: loading cluster ID for follower catalog manager: success
W20260925 05:00:24.525496 29352 catalog_manager.cc:1576] T 00000000000000000000000000000000 P fba2a365442d40c7b44cf58d6a639540: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260925 05:00:24.525734 29352 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260925 05:00:24.526074 29331 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 05:00:24.527729 29331 catalog_manager.cc:1504] Loading token signing keys...
I20260925 05:00:24.569940 29331 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75b663e2c09043658a3ace1ca5558b51: Generated new TSK 0
I20260925 05:00:24.571118 29331 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 05:00:24.724584 29143 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:00:24.705350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48472 (local address 127.6.29.190:43911)
0925 05:00:24.705596 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:24.705603 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:24.722150 (+ 16547us) server_negotiation.cc:407] Connection header received
0925 05:00:24.722778 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:24.722787 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:24.722884 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:24.722975 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:24.724268 (+ 1293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48472: BlockingRecv error: recv got EOF from 127.0.0.1:48472 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 05:00:24.727617 29219 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:00:24.707155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34068 (local address 127.6.29.189:41607)
0925 05:00:24.707388 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:24.707394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:24.725511 (+ 18117us) server_negotiation.cc:407] Connection header received
0925 05:00:24.726088 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:24.726094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:24.726207 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:24.726294 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:24.727329 (+ 1035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34068: BlockingRecv error: recv got EOF from 127.0.0.1:34068 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260925 05:00:24.731594 29329 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:00:24.729099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47682 (local address 127.6.29.188:40077)
0925 05:00:24.729274 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:24.729280 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:24.729774 (+ 494us) server_negotiation.cc:407] Connection header received
0925 05:00:24.730020 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:24.730025 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:24.730176 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:24.730258 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:24.731313 (+ 1055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47682: BlockingRecv error: recv got EOF from 127.0.0.1:47682 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260925 05:00:24.777113 29219 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:00:24.762268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34080 (local address 127.6.29.189:41607)
0925 05:00:24.762477 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:24.762483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:24.774671 (+ 12188us) server_negotiation.cc:407] Connection header received
0925 05:00:24.775086 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:24.775094 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:24.775263 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:24.775438 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:24.776774 (+ 1336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34080: BlockingRecv error: recv got EOF from 127.0.0.1:34080 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260925 05:00:24.830294 29219 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:00:24.814783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34084 (local address 127.6.29.189:41607)
0925 05:00:24.814997 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:24.815005 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:24.828470 (+ 13465us) server_negotiation.cc:407] Connection header received
0925 05:00:24.828740 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:24.828747 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:24.828847 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:24.828938 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:24.830043 (+ 1105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34084: BlockingRecv error: recv got EOF from 127.0.0.1:34084 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260925 05:00:24.871641 29219 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:00:24.869299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34090 (local address 127.6.29.189:41607)
0925 05:00:24.869523 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:24.869532 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:24.870110 (+ 578us) server_negotiation.cc:407] Connection header received
0925 05:00:24.870355 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:24.870359 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:24.870458 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:24.870574 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:24.871367 (+ 793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34090: BlockingRecv error: recv got EOF from 127.0.0.1:34090 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260925 05:00:24.884229 29219 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:00:24.882064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34106 (local address 127.6.29.189:41607)
0925 05:00:24.882257 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:24.882266 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:24.882619 (+ 353us) server_negotiation.cc:407] Connection header received
0925 05:00:24.882849 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:24.882855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:24.882998 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:24.883112 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:24.884000 (+ 888us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34106: BlockingRecv error: recv got EOF from 127.0.0.1:34106 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260925 05:00:24.898072 29219 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:00:24.895822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34118 (local address 127.6.29.189:41607)
0925 05:00:24.896050 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:24.896059 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:24.896623 (+ 564us) server_negotiation.cc:407] Connection header received
0925 05:00:24.896864 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:24.896867 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:24.896942 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:24.897060 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:24.897806 (+ 746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34118: BlockingRecv error: recv got EOF from 127.0.0.1:34118 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260925 05:00:24.920254 29219 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:00:24.917812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34124 (local address 127.6.29.189:41607)
0925 05:00:24.918102 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:24.918112 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:24.918664 (+ 552us) server_negotiation.cc:407] Connection header received
0925 05:00:24.918912 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:24.918916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:24.918991 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:24.919113 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:24.919984 (+ 871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34124: BlockingRecv error: recv got EOF from 127.0.0.1:34124 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260925 05:00:24.959615 29219 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:00:24.957225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34138 (local address 127.6.29.189:41607)
0925 05:00:24.957467 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:24.957477 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:24.958036 (+ 559us) server_negotiation.cc:407] Connection header received
0925 05:00:24.958297 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:24.958302 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:24.958376 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:24.958513 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:24.959312 (+ 799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34138: BlockingRecv error: recv got EOF from 127.0.0.1:34138 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260925 05:00:25.036001 29219 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:00:25.032867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34140 (local address 127.6.29.189:41607)
0925 05:00:25.033346 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:25.033380 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:25.034060 (+ 680us) server_negotiation.cc:407] Connection header received
0925 05:00:25.034307 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:25.034313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:25.034425 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:25.034542 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:25.035648 (+ 1106us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34140: BlockingRecv error: recv got EOF from 127.0.0.1:34140 (error 108)
Metrics: {"server-negotiator.queue_time_us":129}
W20260925 05:00:25.169219 29219 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:00:25.167342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34150 (local address 127.6.29.189:41607)
0925 05:00:25.167545 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:25.167550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:25.167951 (+ 401us) server_negotiation.cc:407] Connection header received
0925 05:00:25.168075 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:25.168079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:25.168170 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:25.168270 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:25.168988 (+ 718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34150: BlockingRecv error: recv got EOF from 127.0.0.1:34150 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 05:00:25.218490 29306 catalog_manager.cc:1259] Loaded cluster ID: 67449104075a4d6d9582cf5d602e8cab
I20260925 05:00:25.218595 29306 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2ad3b8e74ffc47658ba794e55ab8ba20: loading cluster ID for follower catalog manager: success
I20260925 05:00:25.221262 29306 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2ad3b8e74ffc47658ba794e55ab8ba20: acquiring CA information for follower catalog manager: success
I20260925 05:00:25.223101 29306 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2ad3b8e74ffc47658ba794e55ab8ba20: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 05:00:25.434425 29219 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 05:00:25.432505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34154 (local address 127.6.29.189:41607)
0925 05:00:25.432785 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:25.432792 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:25.433070 (+ 278us) server_negotiation.cc:407] Connection header received
0925 05:00:25.433176 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:25.433181 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:25.433258 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:25.433355 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 05:00:25.434224 (+ 869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34154: BlockingRecv error: recv got EOF from 127.0.0.1:34154 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260925 05:00:25.529254 29352 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fba2a365442d40c7b44cf58d6a639540: acquiring CA information for follower catalog manager: success
I20260925 05:00:25.531227 29352 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fba2a365442d40c7b44cf58d6a639540: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 05:00:25.959589 29419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:25.953283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34164 (local address 127.6.29.189:41607)
0925 05:00:25.953821 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:25.953826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:25.953944 (+ 118us) server_negotiation.cc:407] Connection header received
0925 05:00:25.954108 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:25.954112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:25.954295 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:25.954407 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:00:25.954826 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:25.955590 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:25.957231 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:25.957674 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:25.957768 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:25.958326 (+ 558us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:00:25.958873 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:00:25.958991 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:25.959161 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 05:00:25.959234 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":285,"threads_started":1}
I20260925 05:00:25.971841 29425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:25.965893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47686 (local address 127.6.29.188:40077)
0925 05:00:25.966386 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:25.966391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:25.966494 (+ 103us) server_negotiation.cc:407] Connection header received
0925 05:00:25.966654 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:25.966658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:25.966810 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:25.966954 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:00:25.967345 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:25.968166 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:25.969499 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:25.969890 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:25.970011 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:25.970520 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:00:25.971167 (+ 647us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:00:25.971297 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:25.971365 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 05:00:25.971431 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":258,"threads_started":1}
I20260925 05:00:25.987246 29425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:25.979258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47696 (local address 127.6.29.188:40077)
0925 05:00:25.979478 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:25.979487 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:25.980657 (+ 1170us) server_negotiation.cc:407] Connection header received
0925 05:00:25.980948 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:25.980956 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:25.981082 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:25.981192 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:00:25.981814 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:25.982893 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:25.985186 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:25.985914 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:25.986062 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:25.986458 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:00:25.986579 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:00:25.986690 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:25.986840 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 05:00:25.986938 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 05:00:25.988224 29419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:25.978322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34170 (local address 127.6.29.189:41607)
0925 05:00:25.978572 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:25.978581 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:25.980458 (+ 1877us) server_negotiation.cc:407] Connection header received
0925 05:00:25.980704 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:25.980711 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:25.980805 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:25.980896 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:00:25.981461 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:25.982668 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:25.986170 (+ 3502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:25.986718 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:25.986856 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:25.987414 (+ 558us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:00:25.987531 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:00:25.987610 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:25.987829 (+ 219us) server_negotiation.cc:299] Negotiation successful
0925 05:00:25.987927 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 05:00:25.988807 29432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 05:00:25.978196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48484 (local address 127.6.29.190:43911)
0925 05:00:25.981828 (+ 3632us) server_negotiation.cc:206] Beginning negotiation
0925 05:00:25.981835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 05:00:25.981859 (+ 24us) server_negotiation.cc:407] Connection header received
0925 05:00:25.981953 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 05:00:25.981957 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 05:00:25.982118 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 05:00:25.982323 (+ 205us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 05:00:25.983040 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:25.983716 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:25.985608 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 05:00:25.986273 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 05:00:25.986401 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 05:00:25.987331 (+ 930us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 05:00:25.987858 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 05:00:25.988387 (+ 529us) server_negotiation.cc:1035] Waiting for connection context
0925 05:00:25.988438 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 05:00:25.988527 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3502,"thread_start_us":3364,"threads_started":1}
I20260925 05:00:26.000399 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 29074
I20260925 05:00:26.001714 29136 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:00:26.118194 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 29074
I20260925 05:00:26.140177 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 29148
I20260925 05:00:26.141252 29210 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:00:26.255402 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 29148
I20260925 05:00:26.276162 6262 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu pid 29222
I20260925 05:00:26.277344 29284 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 05:00:26.387440 6262 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskjC0Yza/build/asan/bin/kudu with pid 29222
2026-09-25T05:00:26Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5154 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10296 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (372684 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST